NAVAL 

POSTGRADUATE 

SCHOOL 

MONTEREY,  CALIFORNIA 


DISSERTATION 


EVOLVING  A  SIMULATION  MODEL  PRODUCT  LINE 
SOFTWARE  ARCHITECTURE  FROM  HETEROGENEOUS 
MODEL  REPRESENTATIONS 

by 

Kevin  J.  Greaney 
September  2003 

Dissertation  Supervisors:  Luqi 

James  B.  Michael 


Approved  for  public  release;  distribution  is  unlimited 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


|  REPORT  DOCUMENTATION  PAGE 

Form  Approved  OMB  No.  0704-0188  1 

Public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  1  hour  per  response,  including 
the  time  for  reviewing  instruction,  searching  existing  data  sources,  gathering  and  maintaining  the  data  needed,  and 
completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any 
other  aspect  of  this  collection  of  information,  including  suggestions  for  reducing  this  burden,  to  Washington 
headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite 
1204,  Arlington,  VA  22202-4302,  and  to  the  Office  of  Management  and  Budget,  Paperwork  Reduction  Project 
(0704-0188)  Washington  DC  20503. 

1.  AGENCY  USE  ONLY  (Leave  blank)  2.  REPORT  DATE  3.  REPORT  TYPE  AND  DATES  COVERED 

!  September  2003  Dissertation 

4.  title  and  subtitle:  Evolving  a  Simulation  Model  Product 
Line  Software  Architecture  from  Heterogeneous  Model  Repre¬ 
sentations 

5.  FUNDING  NUMBERS 

6.  AUTHOR(S)  Kevin  J.  Greaney 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Naval  Postgraduate  School 

Monterey,  CA  93943-5000 

8.  PERFORMING 

ORGANIZATION  REPORT 
NUMBER 

9.  SPONSORING  /  MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

Missile  Defense  Agency 

Washington,  DC  20301-7100 

10.  SPONSORING  /  MONITORING 
AGENCY  REPORT  NUMBER 

11.  SUPPLEMENTARY  NOTES  The  views  expressed  in  this  thesis  are  those  of  the  author  and  do  not  reflect  the  official 
policy  or  position  of  the  Department  of  Defense  or  the  U.S.  Government. 

12a.  DISTRIBUTION  /  AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  is  unlimited 

12b.  DISTRIBUTION  CODE 

ABSTRACT  (maximum  200  words)  National-  and  Department-level  decision-makers  expect  credible 
Department  of  Defense  models  and  simulations  (M&S)  to  provide  them  confidence  in  the  simulation  results,  espe¬ 
cially  for  mission-critical  and  high-risk  decisions  supporting  National  Security.  Many  of  these  large-scale,  soft¬ 
ware-intensive  simulation  systems  were  autonomously  developed  over  time,  and  subject  to  varying  degrees  of 
funding,  maintenance,  and  life-cycle  management  practices,  resulting  in  heterogeneous  model  representations  and 
data.  Systemic  problems  with  distributed  interoperability  of  these  non-trivial  simulations  in  federations’  persist, 
and  current  techniques,  procedures,  and  tools  have  not  achieved  the  desired  results. 

The  Software  Architecture -Based  Product  Line  for  simulation  model  representations,  employing  Architecture 
Readiness  Levels  presented  in  this  dissertation  provides  an  alternative  methodology.  The  proposed  four-layered 
M&S  software  architecture -based  product  line  model  enables  the  development  of  model  representations  supported 
by  readiness  levels.  Each  layer  reflects  a  division  of  the  software  architecture-based  product  line.  The  layer 
represents  a  horizontal  slice  through  the  architecture  for  organizing  viewpoints  or  views  at  the  same  level  of  ab¬ 
straction  while  the  software  architecture -based  product  line  represents  a  vertical  slice.  A  layer  may  maintain  mul¬ 
tiple  views  and  viewpoints  of  a  software  architecture -based  product  line.  A  Domain  Metadata  Repository  pre¬ 
scribes  the  interaction  between  layers.  We  introduce  the  Domain  Integrated  Product  Development  Team  concept. 

14.  SUBJECT  TERMS  Models  and  Simulations,  software  architecture,  product  lines,  architecture 
description  languages  (ADL),  Extensible  Markup  Language  (XML),  verification,  validation,  readiness 
levels,  interoperability,  heterogeneous  model  representations  and  data,  credibility,  confidence, 
distributed  development 

15.  NUMBER  OF 
PAGES 

558 

16.  PRICE  CODE 

17.  SECURITY  CLASSIFI¬ 
CATION  OF  REPORT 

Unclassified 

18.  SECURITY  CLASSIFICA¬ 
TION  OF  THIS  PAGE 

Unclassified 

19.  SECURITY  CLAS¬ 
SIFICATION  OF  AB¬ 
STRACT 

Unclassified 

20.  LIMITATION 

OF  ABSTRACT 

UL 

NSN  7540-01-280-5500  Standard  Form  298  (Rev.  2-89) 


Prescribed  by  ANSI  Std.  239-18 


1 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


11 


Approved  for  public  release;  distribution  is  unlimited 


EVOLVING  A  SIMULATION  MODEL  PRODUCT  LINE  SOLTWARE  ARCHI¬ 
TECTURE  FROM  HETEROGENEOUS  MODEL  REPRESENTATIONS 

Kevin  J.  Greaney 
Colonel,  United  States  Army 
B. A.,  Northeastern  University,  1974 
M.S.,  Shippensburg  University,  1977 
M.A.,  Webster  University,  1992 

Submitted  in  partial  fulfillment  of  the 
requirements  for  the  degree  of 

DOCTOR  OF  PHILOSOPHY  IN  SOFTWARE  ENGINEERING 

from  the 

NAVAL  POSTGRADUATE  SCHOOL 
September  2003 


Author: 


Approved  by: 


Approved  by: 


Kevin  J.  Greaney 

Luqi 

Professor  of  Computer  Science 
Dissertation  Committee  Chair* 
Dissertation  Co-Advisor 

Valdis  Berzins 

Professor  of  Computer  Science 

James  B.  Michael 

Professor  of  Computer  Science 
Dissertation  Co-Advisor 

John  A.  Hamilton,  Jr., 
Professor  of  Computer  Science 
Auburn  University 

Patricia  A.  Sanders 

System  Executive  Officer 

Missile  Defense  Agency 

Peter  Denning,  Chair,  Department  of  Computer  Science 

Carson  K.  Eoyang,  Associate  Provost  for  Academic  Affairs 


iii 


Approved  by: 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


IV 


ABSTRACT 


National-  and  Department-level  decision-makers  expect  credible  Department  of 
Defense  models  and  simulations  (M&S)  to  provide  them  confidence  in  the  simulation 
results,  especially  for  mission-critical  and  high-risk  decisions  supporting  National  Secu¬ 
rity.  Many  of  these  large-scale,  software-intensive  simulation  systems  were  autono¬ 
mously  developed  over  time,  and  subject  to  varying  degrees  of  funding,  maintenance,  and 
life-cycle  management  practices,  resulting  in  heterogeneous  model  representations  and 
data.  Systemic  problems  with  distributed  interoperability  of  these  non-trivial  simulations 
in  federations’  persist,  and  current  techniques,  procedures,  and  tools  have  not  achieved 
the  desired  results. 

The  Software  Architecture-Based  Product  Line  for  simulation  model  representa¬ 
tions,  employing  Architecture  Readiness  Levels  presented  in  this  dissertation  provides  an 
alternative  methodology.  The  proposed  four-layered  M&S  software  architecture-based 
product  line  model  enables  the  development  of  model  representations  supported  by 
readiness  levels.  Each  layer  reflects  a  division  of  the  software  architecture -based  product 
line.  The  layer  represents  a  horizontal  slice  through  the  architecture  for  organizing  view¬ 
points  or  views  at  the  same  level  of  abstraction  while  the  software  architecture-based 
product  line  represents  a  vertical  slice.  A  layer  may  maintain  multiple  views  and  view¬ 
points  of  a  software  architecture-based  product  line.  A  Domain  Metadata  Repository 
prescribes  the  interaction  between  layers.  We  introduce  the  Domain  Integrated  Product 
Development  Team  concept. 
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I. 


INTRODUCTION 


A.  THE  NEW  STRATEGIC  ROLE  FOR  MODELS  AND  SIMULATIONS 

Software1  enables  the  political  [Bus02d],  economic  [RTI02],  psychological 
[Bus02b,  Bus02c],  and  military  [RumOl]  centers  of  United  States  national  power.  The 
United  States  military  center  of  national  power,  manifested  in  the  Department  of  Defense” 
(DoD),  employs  software-intensive  systems  in  a  wide  variety  of  missions,  tasks,  and  func¬ 
tions,  including  models  and  simulations4  (M&S),  supporting  the  Department’s  national  de¬ 
fense  mission.  The  Department  requires  credible5  M&S  supporting  confidence6  in  the 
simulation  results  provided  to  Department-  and  National-level  decision-makers,  especially 
in  high-risk,  mission-critical  decisions  [DoD94],  Establishing  credibility  in  Department 
simulations  involves  many  disciplines  and  knowledge  areas  including  software  engineer¬ 
ing7,  processes8,  quality9,  product  management10  and  architecture11.  The  Department’s 
complex  organizational  dynamics,  and  complicated  acquisition  procedures  [BSK+00, 
SAG+00,  Wol02a,  Wol02b]  also  impact  the  level  of  M&S  credibility,  at  times  adversely. 

There  is  a  new  sense  of  urgency  in  the  Department  for  credible  M&S,  and  an  in¬ 
creased  emphasis  on  improved  confidence  in  simulation  results  [DoD03a],  In  the  past, 
models  and  simulations  played  secondary  and  tertiary  roles  supporting  the  Department’s 
decision-making  process.  Today,  models  and  simulations  are  Department  corporate  assets 
[DoD94],  evolving  into  primary  enablers  directly  supporting  an  information  age  national 

1  Software  includes  the  computer  programs,  procedures,  and  possibly  associated  documentation  and  data  pertaining  to  the  operation  of  a 

computer  system  [IEE90]. 

2 

Department  of  Defense  will  be  referred  to  by  the  terms  Department  or  DoD,  used  interchangeably  throughout  this  document. 

3  ...  .  . 

A  software-intensive  system  is  any  system  where  software  contributes  essential  influences  to  the  design,  construction,  deployment,  and 
evolution  of  the  system  as  a  whole  [IEEOOb]. 

4 

Models  and  simulations  (M&S)  -  In  this  research  models  and  simulations  (M&S)  refers  to  mathematical  abstractions  and  software 
implementations,  as  opposed  to  the  term  modeling  and  simulations  used  to  identify  an  analytical  problem  solving  approach  [GMS+96]. 

Credible  means  1.  Capable  of  being  believed;  believable;  plausible;  2.  Worthy  of  confidence;  reliable.  Plausible  is  defined  as  1.  Seem¬ 
ingly  or  apparently  valid,  likely,  or  acceptable.  Reliable  is  further  defined  as  1.  That  can  be  relied  upon;  dependable  [Web95]. 

6  Confidence  is  defined  as  1. Trust  or  faith.  2.  A  trusting  relationship.  3.  Something  confided.  4.  A  feeling  of  assurance,  especially  of  self- 
assurance.  5.  The  assurance  that  someone  keeps  a  secret  [Web95]. 

7  ...  .  .  ....  .  ... 

Software  engineering  is  the  application  of  a  systematic,  disciplined,  quantifiable  approach  to  the  development;  operation,  and  mainte¬ 
nance  of  software;  that  is,  the  application  of  engineering  to  software  [IEE90]. 

Process  is  a  sequence  of  steps  performed  for  a  given  purpose:  for  example,  the  software,  development  process  [IEE90]. 

9 

Quality  is  1).  The  degree  to  which  a  system,  component,  or  process  meets  specified  requirements;  2).  The  degree  to  which  a  system, 
component,  or  process  meets  customer  or  user  needs  and  expectations  [IEE90]. 

Product  management  includes  the  definition,  coordination,  and  control  of  the  product  characteristics  during  its  development  cycle 
[IEE90. 

Architecture  means  the  organizational  structure  of  a  system  or  component  [IEE90]. 

l 


security  strategy  [Bus02d].  The  Department’s  support  for  this  strategy  and  supporting  ob¬ 
jectives  requires  confidence,  credibility,  security  and  interoperability  in  M&S,  which  are 
“broadly  applicable  to  the  full  range  and  scope  of  missions  and  activities  across  the  breadth 
of  DoD  operations”  [DoDOla]  and  enable  the  Department’s  execution  of  critical  missions, 
tasks,  and  functions  [Sha97]. 

National-level  decision-makers  require  credible  mission-critical  M&S  software12 
supporting  confidence  in  national  security  policy  decisions  including  the  emerging  Home¬ 
land  Defense  security  policy  [FOP+Ol,  Bre02b,  Bus02a,  Bus02b,  Bus02c,  CS02]  and  com¬ 
plex,  mission-critical,  high-risk,  defensive  systems  such  as  ballistic  missile  defense,  de¬ 
signed  to  counter  asymmetric  twenty-first  century  threats  against  the  American  Homeland, 
including  global  terrorism  and  weapons  of  mass  destruction  [Bus02e].  President  Bush  re¬ 
cently  signed  the  National  Strategy  For  Homeland  Security  [Bus02c]  emphasizing  six  mis¬ 
sion-critical  areas  supporting  the  Nation’s  defense  including  the  protection  of  critical  infra¬ 
structure  and  key  assets.  Within  the  critical  infrastructure  mission  area,  the  President  iden¬ 
tified  eight  major  initiatives  including  the  development  of  a  national  infrastructure  protec¬ 
tion  plan,  securing  cyberspace,  and  harnessing  “the  best  analytic  and  modeling  tools  to  de¬ 
velop  effective  protective  solutions”  [Bus02c]. 

From  a  strategic  perspective,  [DoD02d]  noted,  “Modeling  and  simulation  are  sim¬ 
ply  a  means  to  an  end”  [DoD02d].  Confidence  in  the  high-risk,  mission-critical  results  pro¬ 
duced  from  the  Department’s  simulations  employed  as  a  means  for  National  Security 
necessitate  the  highest  levels  of  credibility.  The  Department’s  draft  Modeling  and  Simula¬ 
tion  Strategic  Plan  [DoD03a]  reflects  this  new  emphasis  on  the  warfighter  with  the  first 
Department  M&S  mission  statement:  “Exploit  Modeling  and  Simulation  to  Transfonn  the 
Warfighting  Capabilities  of  the  United  States”  [DoD03a]. 


12 


Critical  software  is  software  whose  failure  could  have  on  safety,  or  could  cause  large  financial  or  social  loss  [IEE90]. 
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B. 


AUTHORITATIVE  REPRESENTATIONS  AND  CREDIBLE  M&S 
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Credible  M&S  requires  authoritative  representations  ,  according  to  the  Depart¬ 
ment’s  M&S  policy  directives,  DoD  Modeling  and  Simulation  Management  [DoD94]  and 
the  Department’s  Modeling  and  Simulation  Master  Plan  [DoD95],  A  Department  M&S 
baseline  assessment  [DoD95]  identified  authoritative  representations  as  one  of  the  “many 
shortfalls  that  must  be  corrected  to  realize  the  DoD  M&S  vision”14  [DoD95]  and  support 
the  Department’s  readiness,  modernization,  force  structure,  and  sustainability  requirements. 
In  this  research  we  employed  the  concept  of  computer-based  simulations  defined  by 
[CS97]  as  a  wide  range  of  computationally  based  activities  employing  digital  computers. 

In  addition,  we  view  the  Department’s  large-scale,  legacy15  modeling  and  simulation  sys¬ 
tems  as  software-intensive  systems. 

Attesting  to  the  importance  of  credible  M&S  representations  in  Department  M&S, 
three  of  the  six  major  [DoD95]  objectives  addressed  authoritative  representations.  This 
research  focused  on  [DoD95]  objective  three,  authoritative  system16  representations.  Other 
major  [DoD95]  objectives  included  the  natural  environment  authoritative  representations 
(objective  two),  and  human  behavior  authoritative  representations  (objective  four).  Ena¬ 
bling  tasks  and  timelines  established  to  achieve  this  [DOD95]  objective  included: 

i  n 

•  Build  prototype  classes  of  objects  as  part  of  the  architecture  prototype  effort  in 
fiscal  year  1995, 

•  Identify  common  object  classes  in  fiscal  year  1996, 

•  Assign  DoD  executive  agents  to  develop  common  object  classes  in  fiscal  year  1996, 

•  Identify  and  assign  responsibilities  for  developing  system  M&S  for  all  DoD  mission 
areas  by  fiscal  year  1997, 


Authoritative  representations  are  models,  algorithms,  and  data  that  have  been  developed  or  approved  by  a  source  which  have  accurate 

technical  knowledge  of  the  entity  or  phenomenon  to  be  modeled  and  its  effects  [DoD95]. 

14 

Department  M&S  Vision  (1995)  A.  Defense  modeling  and  simulation  will  provide  readily  available,  operationally  valid  environments 
for  use  by  the  DoD  Components:  (1)  To  train  jointly,  develop  doctrine  and  tactics,  formulate  operational  plans,  and  assess  warfighting 
situations.  (2)  To  support  technology  assessment,  system  upgrade,  prototype  and  full-scale  development,  and  force  structuring.  B.  Fur¬ 
thermore,  common  use  of  these  environments  will  promote  a  closer  interaction  between  the  operations  and  acquisition  community  in 
carrying  out  their  respective  duties.  To  allow  maximum  utility  and  flexibility,  these  modeling  and  simulation  environments  will  be  con¬ 
structed  from  affordable,  reusable  components  interoperating  through  an  open-system  architecture  [DoD95]. 

15  Legacy  M&S  -  M&S  systems  developed  in  the  past  without  the  benefit  of  modem  standards,  and  still  in  use  [PMR94]. 

16  DoD  M&S  system  representations  include  weapons,  sensors,  units,  command,  control,  communications,  computers  and  intelligence, 
surveillance,  reconnaissance  (C4ISR)  platforms,  and  logistic  support  for  a  wide  scope  of  U.S.,  Allied,  Coalition,  and  major  threat  plat¬ 
forms  [DoD95]. 

17  ...  . 

Architecture  in  this  context  is  defined  as  the  structure  of  components  in  a  program/system,  their  interrelationships,  and  principles  and 
guidelines  governing  their  design  and  evolution  over  time  [DoD95].  Other  definitions  are  introduced  later. 
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•  Develop  aggregation/disaggregation  methodologies  by  a  date  to  be  determined 
[DoD95]. 

C.  LIMITATIONS  OF  CURRENT  APPROACHES  TO  ACHIEVE  CREDIBIL¬ 
ITY 

The  Defense  Modeling  and  Simulation  Office  (DMSO)  initiated  several  studies  in 
2000  to  assess  the  progress  of  M&S  within  the  Department  since  the  1995  publication  of 
the  M&S  Master  Plan  [DoD95].  The  DMSO  methodology  employed  multiple  techniques 
including  written  surveys  and  interviews  [WriOlb]  with  key  members  of  the  Department’s 
M&S  community,  sponsorship  of  the  Integration  Task  Force  (ITF)  effort,  and  completion 
of  a  Warfighter  M&S  needs  assessment.  These  initiatives  produced  three  major  product 
deliverables,  a  Master  Plan  Baseline  Assessment  [MosOO,  CraOla,  CraOlb],  a  final  assess¬ 
ment  Report  of  Warfighter  M&S  Needs  [MWD+00],  and  the  Report  of  the  Integration  Task 
Force  [HCG+Ola,  HCB+Olb].  The  survey  results  indicated  that  most  of  the  [DoD95]  ob¬ 
jectives  experienced  some  progress,  with  the  greatest  progress  noted  on  the  development  of 
a  common  technical  framework,  and  the  establishment  of  the  High  Level  Architecture 
(HLA),  an  Institute  of  Electrical  and  Electronic  Engineers  (IEEE)  standard  [IEE98e, 
IEEOOc,  IEEOOd,  IEEOOe],  and  the  Department’s  technical  architecture  for  simulation  in¬ 
teroperability  [GanOO], 

However,  most  objectives  for  authoritative  representations  remained  incomplete, 
with  the  least  progress  noted  on  the  representation  of  human  behavior  [MosOO,  MT01, 
Flo02a].  The  three  DMSO  study  initiatives  [HCG+Ola,  HCG+Olb,  MWD+00,  and  MosOO] 
collectively  provided  direction  to  the  Department’s  2001  draft  revision  of  the  DoD  Model¬ 
ing  and  Simulation  Master  Plan  (Draft)  [DoDOla].  Many  of  the  persistent  systemic  prob¬ 
lems  with  authoritative  representations  require  long-term  executable  solutions  to  improve 
quality  and  evidence  that  M&S  provides  a  return  on  investment  [Mye99].  The  true  scope 
of  the  issues,  challenges,  and  difficulties  faced  by  the  Department’s  M&S  community  to 
develop  authoritative  representations  is  evident  by  a  comparison  of  the  system  authoritative 
representation  sub-objectives  of  the  approved  [DoD95]  and  the  draft  [DoDOla]  plan.  Simi- 

1 8  Aggregation  is  the  ability  to  group  entities  while  preserving  the  effects  of  entity  behavior  and  interaction  while  grouped.  Disaggrega¬ 
tion  is  the  ability  to  represent  the  behavior  of  an  aggregated  unit  in  terms  of  its  component  entities  based  on  maintained  state  representa¬ 
tions  [DoD98]. 
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lar  [DoD95]  objectives  originally  scheduled  for  implementation  in  the  1995-1997  time- 
frame,  slipped  almost  a  decade  to  2004-2006  [DoDOla]. 

There  were  several  understandable  reasons  for  the  [DoD95]  schedule  slip.  A  few  of 
the  reasons  included  a  very  ambitious  schedule  negatively  impacted  by  Department  budget 
cuts  during  the  1990s;  the  sheer  scope  of  the  work  across  the  Department,  Service  and 
M&S  domains19;  the  breadth  of  the  technical  challenges,  including  improved  fidelity20  of 
authoritative  representations  [Gro99,  RGH00];  and  the  overall  immaturity  of  software  en¬ 
gineering  processes  and  practices  [HHK93,  HNC+00]  supporting  M&S  development. 

Other  contributing  factors  to  the  [DoD95]  schedule  slip  included  significant  interoperabil¬ 
ity  issues,  external  forces  including  the  Department’s  balky  acquisition  process  [Mos02], 
changing  Department  priorities  [FerOl];  and  the  unplanned  allocation  of  resources  to  Year 
2000  problem  remediation  [Gre97,  WG99].  The  draft  [DoDOla]  again  reiterated  the  need 
for  “credible  authoritative  representations”  (sub-objective  3.3),  which  are  interoperable, 
consistent,  and  promote  “access  to  the  underlying  algorithms  and  data,  which  define  the 
representation  within  M&S  by  other  applications”  [DoDOla]. 

The  Department  also  continues  to  experience  persistent,  systemic  M&S  architecture 
problems.  The  C4ISR  Information  Superiority  Master  Plan  [DoD02d]  identified  continu¬ 
ing  concerns  with  the  Department’s  M&S  technical  architecture  noting  the  HLA  for  De¬ 
partment  M&S  lacked  seamless  interoperability  with  the  Common  Operating  Environment 
(COE)  [JTA02a,  JTA02b]  supporting  the  development  of  the  Global  Command  and  Con¬ 
trol  System  (GCCS)  and  the  Department’s  Command,  Control,  Communications,  Com¬ 
puters,  Intelligence,  Surveillance  and  Reconnaissance  (C4ISR)  architecture  framework 
[AWG97].  The  proposed  [DoD02d]  solution  is  to  develop  a  common  Technical  Reference 
Model  (TRM)  for  the  C4ISR  and  the  M&S  community,  evolving  over  the  next  twenty 


19 

Domains  are  the  physical  or  abstract  space  in  which  the  entities  and  processes  operate.  The  domain  can  be  the  land,  sea,  air,  space, 
undersea,  a  combination  of  any  these,  or  an  abstract  domain  such  as  an  n-dimensional  mathematics  space,  or  economic  or  psychological 
domains  [DoD98]. 

20 

Fidelity:  1 .  The  degree  to  which  a  model  or  simulation  reproduces  the  state  and  behavior  of  a  real  world  object  or  the  perception  of  a 
real  world  object,  feature,  condition,  or  chosen  standard  in  a  measurable  or  perceivable  manner;  a  measure  of  the  realism  of  a  model  or 
simulation;  faithfulness.  Fidelity  should  generally  be  described  with  respect  to  the  measures,  standards  or  perceptions  used  in  assessing 
or  stating  it.  2.  The  methods,  metrics,  and  descriptions  of  models  or  simulations  used  to  compare  those  models  or  simulations  to  their 
real  world  referents  or  to  other  simulations  in  such  terms  as  accuracy,  scope,  resolution,  level  of  detail,  level  of  abstraction  and  repeat¬ 
ability.  Fidelity  can  characterize  the  representations  of  a  model,  a  simulation,  the  data  used  by  a  simulation  (e.g.,  input,  characteristic  or 
parametric),  or  an  exercise.  Each  of  these  fidelity  types  has  different  implications  for  the  applications  that  employ  these  representations 
[Gro99,  RGH00,  RPG00] 
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years.  Based  on  the  actual  completion  dates  of  previous  Department  M&S  projects,  this 
time-to-completion  estimate  for  a  TRM  may  be  accurate.  However,  it  is  not  clear  if  this 
timeline  is  responsive  to  the  Department’s  operational  requirements  and  transfonnation  ini¬ 
tiatives  [MAK01],  since  the  goal  for  the  Department  is  to  have  legacy  systems  interoper¬ 
able,  in  tenns  of  command  and  control  functions,  by  the  end  of  fiscal  year  2008  [WolOl, 
FBK+01]. 

Lessons-leamed  from  incorporating  M&S  into  the  Department’s  test  and  evaluation 
(T&E)  process  [DOTOla,  DOTOlb]  confirmed  the  inherent  limits  of  M&S  cited  twenty 
years  earlier  [PAD78,  PAD80],  [DOTOla,  DOTOlb]  revalidated  the  lack  of  quality,  au¬ 
thoritative  system  representations  as  the  Department’s  chronic  systemic  M&S  shortfall 
with  the  “underlying  issues  of  the  credibility  of  the  tools  [being]  a  major  constraint” 
[PHP+96].  Improving  the  fidelity  ,  accuracy"  and  precision'  quality  attributes  of  au¬ 
thoritative  representations  supports  credibility.  However,  M&S  quality  has  also  been  an 
elusive  objective  for  the  Department.  Many  prior  systemic  deficiencies  continue  to  exist, 
including  the  need  for  authoritative  representations  and  data,  improved  infonnation  assur¬ 
ance  measures  [HGE03],  and  interoperability  standards  [HCG+Ola,  HCG+Olb,  You02c]. 
We  refer  to  the  entire  class  of  problems  impeding  the  development  of  authoritative  repre¬ 
sentations  as  heterogeneous  system  representation  anomalies  [IEE93],  discussed  in  Chapter 
IV. 

Heterogeneous  system  representation  anomalies,  a  major  focus  of  this  study,  in¬ 
clude  two  broad  categories  adversely  affecting  simulation  federation  credibility:  technical 
interoperability  and  substantive  interoperability  [DSB+99,  YPH01].  Technical  interopera¬ 
bility  [YPHO 1  ]  is  the  capability  of  federates  to  physically  connect  and  exchange  data,  while 
substantive  interoperability  [YPH01]  adds  the  requirement  for  the  federation  to  provide 


Fidelity  is  the  accuracy  of  the  representation  when  compared  to  the  real  world  [DoD95]. 

22 

Accuracy:  The  degree  to  which  a  parameter  or  variable  or  set  of  parameters  or  variables  within  a  model  or  simulation  conform  ex¬ 
actly  to  reality  or  to  some  chosen  standard  or  referent.  See  resolution,  fidelity,  precision  [RPG00]. 

23 

Precision:  1 .  The  quality  or  state  of  being  clearly  depicted,  definite,  measured  or  calculated.  2.  A  quality  associated  with  the  spread 
of  data  obtained  in  repetitions  of  an  experiment  as  measured  by  variance;  the  lower  the  variance,  the  higher  the  precision.  3.  A  measure 
of  how  meticulously  or  rigorously  computational  processes  are  described  or  performed  by  a  model  or  simulation  [RPG00]. 

24 

An  attribute  is  a  property  or  characteristic  of  one  or  more  entities,  a  property  inherent  in  an  entity,  or  associated  with  that  entity  for 
database  purposes  [DoD98]. 
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consistent,  accurate  simulated  representations,  adhering  to  “fair  fight”  principles25  to  ade¬ 
quately  meet  federation  accreditation  (e.g.,  intended  use)  objectives.  Substantive  interop¬ 
erability  [Dav98,  DSB+99,  YPH01]  involves  the  representation  validity  supporting  the  fed- 
erations’  intended  use,  interoperability  among  entities'  ,  and  the  contextual  effects  on  in- 

97 

teroperability'  .  The  logical  interaction  of  federation  entities  involves: 

•  The  level  of  representation  for  the  entity  to  include  the  necessary  level  of  detail  and 
goodness-of-fit, 

•  The  key  entity  model  attributes  reflecting  critical  real-world  characteristics  salient 
to  the  federation’s  purpose, 

•  The  entity  models  exhibit  the  needed  behaviors,  work  together  logically,  and  em¬ 
ploy  consistent  algorithms  across  the  federation  to  compute  effects  [YPH01]. 

D.  RESEARCH  QUESTION  AND  HYPOTHESIS 

The  primary  focus  of  this  dissertation  was  to  determine  if  system  representations  in 
five  selected  Missile  Defense  Agency’s  (MDA)  software-intensive  M&S  systems  were 
authoritative  according  to  current  Department  standards,  and  credible  enough  to  support 
confidence  by  Department-  and  National-level  decision-makers  in  missile  defense  deci¬ 
sions  critical  to  national  security. 

The  second  objective  included  the  identification  of  pre-conditions  and  alternatives 
to  improve  and  enhance  the  simulation  credibility  and  confidence  in  the  results  produced 
by  the  Agency’s  M&S  software.  The  basic  research  problem  investigated  during  this  effort 
was: 

How  may  the  Agency  better  define,  develop,  evolve,  manage,  and  maintain  authori¬ 
tative  model  representations  in  the  five  selected  large-scale,  software-intensive  legacy 
simulations"  ,  and  effectively  address  simulation  interoperability  issues  (e.g.,  heterogene¬ 
ous  system  representation  anomalies)  within  existing  constraints  and  pre-conditions  (e.g., 


25 

Two  or  more  simulations  may  be  considered  to  be  in  a  fair  fight  when  differences’  in  the  simulations’  performance  characteristics 
have  significantly  less  effect  on  the  outcome  of  the  conflict  than  actions  taken  by  the  simulation  participants  [DoD98]. 

Interoperability  among  entities  includes  accurate  temporal  resolution,  spatial  resolution  and  the  logical  interaction  among  entities 

modeled  in  different  federates  [YPH01]. 

27 

Contextual  effects  on  interoperability  address  the  accurate,  coherent  relationship  required  between  components  of  the  physical  envi¬ 
ronment  to  include  ocean,  ground,  space,  atmosphere,  weather,  infrared  and  electromagnetic  spectrums  [HYOla]. 

28  Hereafter,  the  Missile  Defense  Agency  maybe  referred  to  as  the  Agency  or  the  Ballistic  Missile  Defense  System  (BMDS). 

29 

Chapter  IX  provides  the  analysis  of  the  five  selected  Agency  simulations  under  study. 
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technical,  organizational,  managerial)  affecting  credibility  in  the  five  selected  simulations 
systems? 

In  response  to  this  question  I  submit  the  following  hypothesis:  Employing  a  do- 

30  31  32 

main  -managed,  four-layered  simulation  software  architecture-based  ,  product  line 
model  with  a  referent  layer,  developed  in  a  distributed  domain  integrated  software  engi¬ 
neering  environment  supporting  the  evolution  of  live  selected  Agency  legacy  simulations 
can  improve  the  authority  of  representations  in  the  live  selected  missile  defense  simula¬ 
tions.  The  Software  Architecture-Based  Product  Line  Model  developed  in  a  distributed 
development  environment  by  a  Domain  Integrated  Product  Development  Team  employing 
Architecture  Readiness  Levels  (ARL)  to  measure  quality  provides  such  a  methodology. 

The  methodology  includes  software  development  practice  areas  [JS02] :  software  engineer- 
ing  ,  organizational  management  and  technical  management  [CN02,  Nor03]  compo¬ 
nents  of  the  Department’s  system  engineering  process  [DAU01].  Hypothesis  testing  in¬ 
cludes  concurrent  implementation  of  the  techniques  and  procedures  in  the  Agency’s  M&S 
domain. 

E.  SIMULATION  SOFTWARE  ARCHITECTURE  (SSA)  OVERVIEW 

This  dissertation  introduces  a  four-layer,  simulation  software  architecture-based^ 
product  line  model,  and  architecture  readiness  levels  supporting  distributed  simulation 
software  development  by  a  domain  integrated  product  development  teams  employing  prod- 


30 

Domain  for  the  purpose  of  this  research  is  an  area  of  knowledge  or  activity  characterized  by  a  set  of  concepts  and  terminology  under¬ 
stood  by  practitioners  in  that  area  [Nor03].  A  more  M&S  domain-unique  definition  for  domain  is  provided  in  [DoD98]. 

31  .  .  .  . 

Software  architecture  is  the  fundamental  organization  of  a  system,  embodied  in  its  components,  their  relationships  to  each  other  and 

the  environment,  and  the  principles  governing  its  design  and  evolution  [IEEOOb]. 

32 

A  product  line  is  a  group  of  products  sharing  a  common,  managed  set  of  features  that  satisfy  specific  needs  of  selected  market  or 

mission  [BFG+00]. 

33 

1 .  A  referent  is  a  codified  body  of  knowledge  about  a  thing  being  simulated  [RPG00].  2.  Something  referenced  or  singled  out  for 

attention,  a  designated  object,  real  or  imaginary  or  any  class  of  such  objects  [Gro99,  RGH00]. 

34 

Software  engineering  practice  areas  are  those  practices  necessary  to  apply  the  appropriate  technology  to  create  and  evolve  both  core 

assets  and  products  [JS02]. 

35 

Organizational  management  refers  to  the  management  of  the  business  issues  that  are  visible  at  the  enterprise  level,  as  opposed  to  the 
project  level  [JS02]. 

Technical  management  practices  areas  include  those  management  practices  necessary  to  engineer  the  development  and  evolution  of 

core  assets  and  products  [JS02]. 

37 

Architecture-based  development  is  a  process  that  utilizes  the  software  architecture  as  the  primary  tool  for  the  design,  evolution,  im¬ 
plementation,  management,  migration,  and  understanding  of  a  software  system.  It  involves  organizing  the  work  products  around  the 
architecture;  implementing  the  software  system  based  on  the  architecture;  and  maintaining  the  implementation  to  reflect  changes  in,  and 
to  ensure  conformance  to,  the  architecture  [CN96]. 
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uct  line  practice  areas  to  evolve  Agency’s  the  five  legacy  simulations  heterogeneously  or- 
ganized  in  a  joint  M&S  hierarchy  today.  The  proposed  four-layered  simulation  soft¬ 
ware  architecture -based  product  line  model  supported  by  a  domain  metadata  repository  en¬ 
ables  the  development  of  authoritative  representations  supported  by  readiness  levels40. 

The  four-level  layered  software  architecture-based  product  line  model  replaces  the 
level  of  abstraction41  prescribed  previously  in  the  traditional  Department  M&S  hierarchy42 
(Figure  2-2),  with  each  layer  (e.g.,  referent,  conceptual,  component,  implementation)  re¬ 
flecting  a  division  of  the  software  architecture-based  product  line  model  into  units.  The 
layer  represents  a  horizontal  slice  through  the  architecture  for  organizing  classifiers  or 
packages  at  the  same  level  of  abstraction  [UML99,  MB02,  Mil02b],  while  the  software  ar¬ 
chitecture-based  product  line  represents  a  vertical  slice.  Vertical  slice  components  of  the 
model  include  a  Simulation  Software  Architectural  Framework  (SSAF)  and  the  Simulation 
Product  Line  Architecture  (SPLA).  A  layering  scheme  prescribes  the  interaction  between 
layers.  A  Referent  layer  maintains  multiple  real-world  views43  and  viewpoints44  for  estab¬ 
lishing  the  foundation  of  authoritative  model  representations. 

The  methodology  supports  development  of  software  architecture-based  product  line 
readiness  levels  or  Architecture  Readiness  Levels  (ARL)  and  a  domain  metadata  reposi¬ 
tory,  addressed  in  Chapter  X.  The  ARL  methodology  supports  conceptual  model  valida¬ 
tion,  data  management,  the  verification,  validation  process,  and  the  implementation  of  qual¬ 
ity  attributes  instantiated  in  the  software  implementation.  The  ARL  methodology  also  sup- 


38  Joint  M&S  includes  representations  of  joint  and  service  forces,  capabilities,  equipment,  materiel,  and  services  used  by  the  joint  com¬ 
munity  or  by  two,  or  more,  Military  Services  [DoD95]. 

39 

Hierarchy  is  a  ranking  or  ordering  of  abstractions  [DoD98]. 

40 

Readiness  Levels  are  based  on  Technology  Readiness  Levels  -  a  systemic  metric/measurement  system  developed  by  NASA  that  sup¬ 
ports  assessments  of  the  maturity  of  a  particular  technology  and  the  consistent  comparison  between  different  types  of  technology 
[Man95]. 

41  ...  . 

Abstraction  is  a  view  of  an  object  that  focuses  in  the  information  relevant  to  a  particular  purpose  and  ignores  the  reminder  of  the  in¬ 
formation  [IEE90]. 

42  We  address  simulation  hierarchies  in  Chapter  II.  Originally  devised  to  manage  simulation  complexity,  multiple  viewpoints  of  simu¬ 
lation  hierarchies  evolved  in  many  forms.  The  Department  selected  a  version  of  the  hierarchy  cited  in  Figure  2-2.  Systemic  issues  per¬ 
sist  with  this  simulation  hierarchy  model,  and  the  lack  of  integration  between  the  different  layers  of  the  simulation  hierarchy.  [WGB+03] 

most  recently  addressed  this  systemic  shortcoming  of  the  Department’s  M&S  architecture. 

43 

In  the  conceptual  framework  of  IEEE  Std  1471-2000,  an  architectural  description  (AD)  is  organized  into  one  or  more  constituents 
identified  as  architectural  views.  A  view  addresses  one  or  more  concerns  identified  by  system  stakeholders.  The  term  view  is  used  to 
refer  to  the  expression  of  the  systems  architecture  with  respect  to  a  particular  viewpoint.  A  view  may  consist  of  one  or  more  architectural 

model  [IEEOOb]. 

44  ....  . 

The  viewpoint  is  a  specification  of  the  conventions  for  constructing  and  using  a  view,  a  pattern  or  template  from  which  to  develop 
individual  views  by  establishing  the  purposes  and  audience  for  a  view  and  the  techniques  for  its  creation  and  analysis.  The  viewpoint 
determines  the  language  (including  notation,  model,  or  product  types)  to  be  used  to  describe  the  view,  and  any  associated  modeling 
methods  or  analysis  techniques  applied  to  representations  of  the  view  [IEEOOb]. 
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ports  the  development  of  quantitative  assessments  of  system  representations  by  the  devel¬ 
oper  and  enables  the  measurement  of  the  production  process.  The  domain  metadata  reposi¬ 
tory  supports  the  domain  metadata  registry  supporting  interoperability  with  other  Govern¬ 
ment  agencies  and  private  sector  metadata  registries. 

The  centrally-managed,  distributed,  domain  integrated  product  development  team 
supports  the  second  study  objective:  a  distributed  M&S  software  engineering  management 
methodology  integrating  the  “production-related”  approach  to  attain  high  levels  of  quality 
by  defining  specific  characteristics  of  a  product  with  precise  measurements,  supported  by 
the  “process-related”45  approach  to  prevent  faults  and  minimize  rework.  The  production 
related  approach  employs  a  distributed  architecture-based  software  development  model 
based  on  product  lines. 

The  target  audiences  for  the  software  architecture-based  product  line  family  include 
Departmental-  and  National-level  decision-makers.  Other  key  stakeholders  supported  by 
this  concept  include  the  Department’s  M&S  community;  simulation  software  developers; 
supported  analysis,  training,  acquisition,  experimentation,  and  operational  communities; 
developmental  and  operational  testers;  and  the  private  sector  supporting  M&S  software  en¬ 
gineering,  transformation  initiatives,  and  simulation-based  acquisition  (SBA). 

F.  RESEARCH  CONTRIBUTIONS 

The  Department  seeks  improved  credibility  in  M&S  supporting  Departmental-  and 
National-level  decision-makers’  confidence  in  simulation  results.  Major  issues  with  the 
implementation  of  the  Department’s  verification,  validation,  and  accreditation  (VV&A) 
process,  conceptual  model  development  practices,  software  development  maturity,  process 
improvement,  and  M&S  quality  have  caused  Department  decision-makers  to  question  the 
amount  of  confidence  they  should  place  in  results  produced  by  the  Department’s  M&S.  As 
a  result  is  has  been  difficult  for  the  Department  to  identify  and  justify  a  return  on  invest- 


45 

Acquisition  of  software-intensive  systems  shall  use  process  improvement  and  performance  measures  [DoD03b]. 
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ment  for  M&S,  especially  for  large-scale  M&S  costing  hundreds  of  million  dollars  or  even 
exceeding  a  billion  dollars46,  to  develop  and  maintain  [BGK+00]. 

Conversely,  the  demand  for  improved  M&S  capability,  interoperability,  reusability, 
and  affordability  continue  to  grow  [DoD03a],  driven  by  the  need  to  transform  United  States 
warfighting  capability.  The  main  contributions  of  this  work  add  to  the  model  and  simula¬ 
tion  body  of  knowledge  and  address  a  causal  factor  of  the  credibility  problem  with  the 
definition  and  development  of  a  flexible,  extensible  methodology  for  creating  software  ar¬ 
chitecture-based  product  models  supporting  the  development  of  authoritative  and  consistent 
M&S  representations  in  very  large-scale  software-intensive  M&S  systems. 

[RGHOO]  noted  that  researchers  study  fidelity  and  conceptual  models  separately  and 
independently,  although  they  share  many  similarities  and  have  a  common  source  of  diffi¬ 
culties  with  a  lack  of  formal,  fundamental  simulation  framework.  [RGHOO]  also  suggested 
a  cooperative  research  effort  to  understand  the  mutual  relationship  between  fidelity  and 
conceptual  models  to  resolve  the  common  problems.  The  software  architecture-based 
product  line47  referent  view  incorporates  the  characteristics  and  attributes  of  a  conceptual 
model  domain  architecture  framework.  This  approach  provides  a  method  for  the  evolution 
of  a  domain  conceptual  model  referent  from  an  existing  heterogeneous  Department  M&S 
hierarchy,  and  establishing  a  quantifiable  methodology  for  establishing  M&S  credibility 
not  accomplished  by  previous  methods.  Key  elements  include: 

•  Development  of  a  layered  M&S  software  architecture  supporting  architecture-based 
systems  for  improved  interoperability,  involving  reverse-engineering,  reuse,  and  re¬ 
engineering  techniques, 

•  Identification  of  a  method  to  evolve  an  existing  heterogeneous  M&S  domain  hierar¬ 
chy  into  a  software  architecture-based  product  line  model, 

•  Contributing  to  the  development  of  scalable  solutions  for  building  credible  M&S 
software  solutions, 

•  Supporting  the  development  of  an  alternative  M&S  framework  supporting  verifica¬ 
tion  and  validation  employing  Architecture  Readiness  Levels, 


46 

The  Department’s  Selected  Acquisition  Report  (SAR)  summary  as  of  June  2002  identified  the  cost  estimate  for  the  JSIMS  program  at 

$1.29  billion. 

47 

A  product  line  is  a  group  of  products  sharing  a  common,  managed  set  of  features  that  satisfy  specific  needs  of  selected  market  or 
mission  [BFG+00]. 
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•  Contributing  to  the  development  of  data  quality  measures,  improved  M&S  meta¬ 
data49,  and  the  body  of  knowledge  to  standardize  data  in  support  of  common,  con¬ 
sistent  authoritative  representations; 

•  Adding  to  the  Department’s  Modeling  and  Simulation  Body  of  Knowledge. 

G.  SIGNIFICANCE  OF  DISSERTATIONS  CONTRIBUTIONS 

There  are  significant  Department  M&S  issues  to  resolve,  including  heterogeneous 
system  representation  anomalies.  The  Department  has  yet  to  achieve  the  full  potential  of 
M&S  technology  due  to  a  lack  of  credibility  in  the  current  M&S  practices  including  con¬ 
ceptual  models,  the  VV&A  process,  and  model  quality  attributes  such  as  fidelity  [YPEOO]. 
Simulation  interoperability  has  also  posed  significant  challenges.  All  previous  M&S  inter¬ 
face  protocols50  and  supporting  distributed  interoperability  (e.g.,  ALSP,  DIS,  HLA)  en¬ 
countered  heterogeneous  (e.g.,  substantive  interoperability)  anomalies  and  failed  to  resolve 
the  problem. 

Interim  solutions  such  as  translators  [You02c]  may  provide  near-term  relief,  how¬ 
ever,  they  introduce  other  maintenance,  cost,  performance,  and  accuracy  risks.  In  addition, 
integrating  translators  into  simulation  federations  presents  new  challenges  including  exces¬ 
sive  latency,  or  incorrect  translations  due  to  syntactical  or  semantic  errors.  [TH03]  con¬ 
tends  that  improving  several  different  aspects  such  as  standards,  architectures,  components, 
data  models,  and  processes  may  improve  interoperability  discussed  in  this  research. 

The  strategy  to  replace  many  of  the  large-scale  legacy  M&S  with  new,  better,  soft¬ 
ware  engineered  M&S  solutions  such  as  the  Joint  Simulation  System  (JSIMS),  Joint  War¬ 
fare  Simulation  (JWARS)  [BBG+99d],  and  Joint  Model  and  Simulation  System  (JMASS), 
discussed  in  Chapter  VII,  have  not  achieved  the  desired  results.  All  three  major  Depart¬ 
ment  simulation  software  development  efforts  to  date  exceeded  initial  cost  estimates,  failed 
to  meet  schedule,  and  produced  initial  results  well  short  of  the  design  specifications.  The 


48 

Data  quality  is  the  correctness,  timeliness,  accuracy,  completeness,  relevance  and  accessibility  that  make  data  appropriate  for  use. 
Quality  statements  are  required  for  source,  accuracy  (positional  and  attribute),  currency,  logical  consistency,  completeness  (feature  and 
attribute),  clipping  indicator,  security  classification,  and  releasibility  [DoD98]. 

49 

Metadata  is  data  that  describes  data  and  my  include  definitions,  classification,  accuracy,  data  type,  precision,  and  source  [DoD95]. 
Protocols  are  a  set  of  rules  and  formats,  semantic  and  syntactic,  which  determine  the  communication  behavior  of  simulation  applica¬ 
tions  [DoD95]. 
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introduction  of  a  software  architecture-based  product  line  model  into  the  Department’s 
M&S  domain  has  the  potential  to  support: 

•  A  Department  M&S  Software  Product  Line  Family  Strategy  including  the  devel¬ 
opment  of  an  integrated  architectural  description51  containing  architecture  views, 
components,  and  data  models,  supporting  improved  software  development  of  simu¬ 
lations  with  authoritative  representations, 

•  An  extension  to  the  Department’s  architectural  framework  to  support  a  simulation 
software  architecture -based  product  line  model, 

•  The  development  of  a  software  architecture-based  product-line  simulation  metadata 
registries, 

•  Additions  to  the  Department’s  M&S  Body  of  Knowledge  (MSBOK). 

•  The  evolution  and  maturation  of  product  line  practices  in  the  Department  support¬ 
ing  improved  M&S  software  engineering  processes. 

•  Follow-on  efforts  to  develop  a  composable  M&S  capability  in  the  Department. 

H.  DISSERTATION  VISION 

A  critical  early  decision  in  this  dissertation  effort  involved  whether  to  follow  a 
broad  or  narrow  dissertation  vision.  The  objectives  of  the  NPS  distance  learning  Ph.D.  pro- 
gram  ”  [NPS02]  and  the  scope  of  the  literature  search  cited  in  Chapter  II  established  the 
framework  for  our  decision  process.  There  were  pros  and  cons  for  each  approach.  A  nar¬ 
row  dissertation  vision  supported  a  research  area  concerning  some  specific  aspect  of  the 
Department’s  simulation  software  practice  affecting  simulation  credibility.  The  narrow 
vision  study  approach  potentially  provides  more  focus,  clarity,  and  precision.  However, 
there  are  many  past  and  present  research  efforts  employing  the  narrow  vision  study-scope 
efforts  cited  in  Chapter  II.  Conversely,  there  is  a  dearth  of  broad  vision  research  efforts 
“related  to  the  development  and  evolution  of  complex  [Department  simulation]  software 
systems”  [NPS02],  indicating  a  need  for  a  broad  vision  or  strategic  level  dissertation  vision. 

We  chose  a  broad  dissertation  vision.  Two  historical  precedents  reinforce  this  deci¬ 
sion,  the  first  being  the  concept  of  a  “silver  bullet”  solution  to  the  software  industry’s  woes, 
which  languished  on  for  over  twenty  years  before  private  sector  and  government  executive 


Every  system  has  an  architecture,  and  an  architecture  can  be  recorded  by  an  architectural  description  (AD).  IEEE  Std  1471-2000 
distinguishes  the  architecture  of  a  system,  which  is  conceptual,  from  particular  descriptions  of  that  architecture,  which  are  artifacts  or 
concrete  products  [IEEOOb]. 

52 

The  Ph.D.  program  in  Software  Engineering  is  specifically  designed  for  DoD  software  practitioners  who  want  to  acquire  the  skill  and 
knowledge  to  perform  state-of-the  art  research  on  issues  related  to  the  development  of  large  complex  software  systems,  and  to  intelli¬ 
gently  manage  the  research  of  other  software  practitioners  [NPS02] 
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management  realized  the  software  quality  problem  was  multi-dimensional.  The  lack  of 
credibility  in  Department  M&S  software  is  also  multi-dimensional. 

The  second  historical  precedent  involves  the  industry’s  learning  curve  over  the  past 
fifteen  years  from  the  less-than-successful  “small-grain”  reuse  programs  through  today’s 
“large-grain”  reuse  initiatives,  including  software  product  lines,  a  central  focus  of  this  re¬ 
search.  A  broad  dissertation  vision  requires  an  understanding  of  the  primary  and  contribut¬ 
ing  factors  eroding  credibility  in  Department  simulations  including  organizational  issues, 
quality  models,  software  engineering,  life-cycle  models  and  processes,  software  architec¬ 
ture,  interoperability,  and  information  assurance,  to  name  a  few. 

Another  factor  contributing  to  the  decision  to  pursue  a  broad  dissertation  vision  is  a 
personal  objective  to  add  to  the  software  engineering  and  software  architecture  body  of 
knowledge  to  develop  the  needed  skill  sets  for  the  future  Department  Domain  and  Enter¬ 
prise  Software  Engineers  and  Architects.  It  is  an  interesting  note  that  many  models  in  dif¬ 
ferent  domains  have  five  levels  of  maturity:  Maslow’s  Hierarchy  of  Needs,  the  Capability 
Maturity  Model  family,  the  Cognitive  Hierarchy,  and  the  Infonnation  Superiority  Correla¬ 
tion  Model  [Gre97].  An  additional  model,  the  Levels  of  Infonnation  Systems  Interopera¬ 
bility  (LISI)  [TLW+00]  model  also  has  five  levels  identifying  the  levels  of  interoperability 
maturity  (e.g.,  isolated,  connected,  functional,  domain,  enterprise). 

A  basic  premise  of  each  of  these  models  is  that  the  next  higher  layer  builds  on  con¬ 
tributions  of  the  previous  layer(s).  This  suggests  that  if  the  Department  requires  future 
Domain  and  Enterprise  Level  Software  Engineers  and  Architects,  a  maturation  process  and 
a  multi-dimensional  view  and  understanding  of  the  contributing  knowledge  bases  (e.g. 
software  engineering  or  simulation  body  of  knowledge)  is  in  order. 

The  last  factor  supporting  this  view  is  time.  The  Department’s  Trans  formation 
process  has  substance,  and  the  pace  quickens  daily  in  all  areas  of  the  Department  to  discard 
non-productive  practices  or  systems,  better  integrate  warfighting  capabilities,  introduce  ef¬ 
fective  and  efficient  business  practices,  reduce  major  headquarter  overhead,  and  better  allo¬ 
cate  the  Department’s  human  resources.  Credible  and  timely  modeling  and  simulation  ca¬ 
pabilities  must  quickly  evolve  to  support  these  many  initiatives  or  risk  becoming  irrelevant. 
Three  recent  case  studies  reinforce  this  assessment: 
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•  The  Department  completely  revamped  and  revised  the  non-responsive  acquisition 
procedures  three  times  in  approximately  one  year:  5  April  2002  [ASC02b,  ASC02c, 
DoD02b,  DoD02f],  30  October  2002  [Wol02a,  Wol02b,  DoD02g,  DoD02h],  and  12 
May  2003  [DoD03b,  DoD03c].  This  pace  of  chance  is  unheard  of  in  the  Depart¬ 
ment’s  acquisition  bureaucracy.  Periodic  Department-level  studies,  Defense  Sci¬ 
ence  Boards,  and  Blue  Ribbon  panels  studied  the  systemic  acquisition  problems 
many  times  over  the  past  thirty  years  and  generally  provided  similar  recommenda¬ 
tions  each  time.  For  many  reasons  the  Department  never  enacted  most  of  the  far- 
reaching  recommendations. 

•  The  recent  war  in  Iraq  reflected  this  new  Transformation  thought  process  in  many 
ways  including  vastly  improved  sensor-to-shooter  decision  cycle  times,  improved 
management  and  visibility  of  logistics,  and  a  much  closer  integration  of  all  Service 
warfighting  efforts  enabled  by  information  technology  [OBB+03}. 

•  A  recent  Department  manpower  transformation  initiative  proposes  radical  changes 
in  both  the  military  and  civilian  management  practices  of  the  Department.  The  pro¬ 
posed  civilian  management  changes  affect  bureaucratic  Civil  Service  rules,  an  area 
historically  immune  to  change.  [Hay03a]. 

Time  is  a  critical  factor.  The  Nation’s  strategic  decision  cycle  time  against  foreign 
and  domestic  threats  continues  to  decrease,  while  new  challenges  appear  every  day  to  test 
all  aspects  of  national  power,  including  the  economic,  military,  political,  and  psychological 
pillars.  A  broad  vision,  strategic  view  of  simulation  credibility  supports  the  Department’s 
transfonnation  initiatives,  focusing  on  M&S  as  a  critically  needed  corporate  asset  which 
has  yet  to  achieve  its  full  potential.  The  “silver-bullet”  solution  languished  nearly  twenty 
years  until  discredited  as  a  flawed  concept.  The  challenge  involves  improving  simulation 
credibility  supporting  confidence  in  national  defense  decisions,  while  concurrently  reduc¬ 
ing  the  development  time  and  complexity  for  new  portable  and  distributed  simulations  to 
meet  unknown  future  exigencies.  The  problem  definition  suggests  a  need  for  broad  re¬ 
search  approach  and  evaluation  of  all  germane  factors. 

I.  DISSERTATION  ORGANIZATION 

We  adopted  a  dual-track  research  methodology,  conducting  an  exhaustive  secon¬ 
dary  source  search  to  determine  the  scope  of  the  problem,  complemented  by  primary 
source  research  of  simulation  model  representations  in  a  specific  Department  domain. 
Chapter  II  provides  a  literature  survey  on  the  current  research  areas  relevant  to  Department 
simulation  credibility. 
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The  literature  survey  clearly  reveals  two  major  eras  affecting  Department  simula¬ 
tion  credibility:  (1)  the  Growth  Era  before  1994,  and  (2)  the  Managed  Era,  or  post- 1994 
period,  when  the  Department  began  to  plan,  develop  policy  and  procedures,  budget,  and 
corporately  manage  the  M&S  portfolio  developed  during  the  Growth  Era  and  plan  for  new 
joint-simulations  to  replace  many  of  the  legacy  Growth  Era  M&S.  This  chapter  also  intro¬ 
duces  the  Department’s  M&S  management  framework  for  establishing  credible  simula¬ 
tions,  including  classes  of  simulation,  simulation  hierarchies,  initiatives  to  establish  distrib¬ 
uted  simulation  capabilities,  the  expanding  need  of  credibility  by  the  major  stakeholders 
influencing  the  Department’s  M&S  management  priorities:  acquisition,  training,  opera¬ 
tions,  analysis,  and  experimental  domains. 

Research  indicates  that  M&S  credibility  is  a  multi-dimensioned  problem.  Chapter 
III  through  Chapter  VII  review  different  dimensions  of  the  Department’s  M&S  credibility 
challenge.  A  major  objective  of  this  work  is  to  provide  a  contribution  to  the  next  period  of 
Department  M&S  development — the  Software  Architecture -based  Composable  era. 

Chapter  III  introduces  M&S  credibility  issues  and  identifies  concerns  that  decision¬ 
makers  have  with  Department  simulation  results,  followed  by  an  overview  of  the  Depart¬ 
ments  VV&A  processes,  and  several  factors  contributing  to  the  implementation  of,  or 
shortfalls  of  the  VV&A  processes  in  the  Department.  Two  major  periods,  the  Growth  and 
the  Managed  Eras  characterize  the  efforts  to  improve  credibility  in  the  Department’s  simu¬ 
lation  results. 

Chapter  IV  reviews  several  heterogeneous  challenges  to  Department  M&S  credibil¬ 
ity.  Improving  the  credibility  of  large-scale,  legacy  Department  M&S  requires  an  under¬ 
standing  of  the  underlying  systemic  causes  for  heterogeneous  system  representation 
anomalies,  especially  in  federation  interoperability.  These  diverse  challenges  include  syn¬ 
tactic  and  semantic  heterogeneity,  data  complexity,  interoperability,  technical  and  substan¬ 
tive  interoperability,  fidelity,  and  multiresolution  modeling. 

Chapter  V  provides  the  Department’s  current  architectural  framework  for  resolving 
the  underlying  systemic  causes  generating  the  pre-conditions  for  heterogeneous  system  rep¬ 
resentation  anomalies,  especially  in  federation  interoperability.  The  Department’s  initia¬ 
tives  to  improve  the  credibility  of  large-scale,  legacy  Department  M&S  supporting  distrib- 
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uted  interoperability  include  the  “as-is”  architecture  and  the  evolving  “to-be”  architecture: 
the  Joint  Technical  Architecture,  and  the  Common  Technical  Framework.  The  Common 
Technical  Framework  components  include  the  High-Level  Architecture,  conceptual  model 
requirements,  and  data  standards  supporting  credible  simulation  development. 

Chapter  VI  addresses  software  and  simulation  quality  attributes  affecting  heteroge¬ 
neous  system  representation  anomalies  and  credibility  in  Department  M&S,  especially  in 
federation  interoperability.  The  chapter  reviews  the  internal  and  external  attributes  of  the 
ISO  9126-1  quality  model  [ISOOl],  testing  for  quality  attributes,  and  M&S  quality  attrib¬ 
utes  addressed  in  much  of  the  current  literature,  including  an  overview  of  aggregation  and 
disaggregation.  The  chapter  concludes  with  a  discussion  on  M&S  quality  approaches. 

Chapter  VII  reviews  three  selected  areas  influencing  credibility  of  large-scale,  leg¬ 
acy  simulations  in  the  Department  including  process  improvement,  the  status  of  new  major 
simulation  software  engineering  initiatives  (e.g.,  JWARS,  JSIMS,  JMASS)  and  the  grow¬ 
ing  software  engineering  body  of  knowledge.  The  chapter  also  includes  a  discussion  of  the 
challenges  generated  by  Department  contract  management  oversight  for  achieving  quality 
software  and  reducing  heterogeneous  system  representation  anomalies,  Department  institu¬ 
tional  factors  affecting  M&S  credibility,  and  recent  Department  software  engineering  edu¬ 
cation  initiatives.  In  addition,  research  reveals  a  growing  awareness  that  the  Department 
needs  qualified  software  engineers  who  understand  the  foundations  of  the  software¬ 
intensive  systems  upon  which  we  basing  our  future  security,  economic  prosperity,  and  mili¬ 
tary  preparedness,  including  credible  M&S. 

Chapter  VIII  discusses  traditional  product  lines,  and  introduces  software  product 
lines  and  software  architectures.  The  chapter  also  reviews  core  asset  development,  reverse 
engineering  to  develop  core  assets,  reengineering  core  assets,  component  development  sup¬ 
porting  a  product  line  methodology,  product  line  practice  areas,  and  an  overview  of 
evolving  software  architecture  theory  potentially  applicable  to  improved  M&S  credibility 
and  reducing  heterogeneous  system  representation  anomalies. 

Chapter  IX  provides  an  analysis  of  heterogeneous  system  representation  anomalies 
in  five  selected  M&S  in  a  component  of  the  Nation’s  ballistic  missile  defense  domain,  the 
Missile  Defense  Agency.  The  analysis  reviewed  the  following  areas  affecting  credibility  of 

17 


the  five  systems  under  study:  organizational  change  impacts,  the  National  Security-based 
need  for  credible  M&S,  risk  identification,  a  domain  M&S  overview  to  provide  a  context 
for  the  five  systems  under  study,  an  overview  of  the  five  systems  under  study,  and  an 
analysis  of  selected  model  representations  in  the  five  systems  under  study.  Specific  analy¬ 
sis  areas  affecting  heterogeneous  system  representation  anomalies  and  credibility  included 
VV&A,  fidelity,  M&S  development  demographics,  and  interoperability. 

Chapter  X  introduces  the  detailed  research  design  for  the  architecture-based  product 
line  model  including  the  method,  specification,  design,  and  implementation  of  a  software 
architecture-based  development  methodology  for  a  product  line  referent  with  multiple 
views  and  viewpoints.  The  chapter  also  introduces  the  M&S  architecture-based  product 
line  model  as  an  abstract  software  architecture -based  foundation,  supported  by  Architecture 
Readiness  Levels  (ARL),  for  resolving  heterogeneous  system  representation  anomalies,  and 
improving  credibility  in  federation  interoperability. 

Chapter  XI  presents  the  major  conclusions  of  this  research,  suggests  areas  for  future 
research,  and  provides  the  significance  of  a  software  architecture-based  product  line  model 
supporting  M&S  credibility.  We  identify  possible  future  research  opportunities  for  the 
software  architecture-based  product  line  model  methodology  not  addressed  within  the 
scope  of  this  research  (e.g.,  composability,  managing  product  line  variability). 
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II.  LITERATURE  SEARCH  AND  DOMAIN  M&S  BACKGROUND 


A.  INTRODUCTION 

Chapter  II  provides  a  literature  survey  on  the  current  research  areas  relevant  to  De¬ 
partment  simulation  credibility.  The  literature  survey  clearly  reveals  two  major  eras  affect¬ 
ing  Department  simulation  credibility:  (1)  the  Growth  Era  before  1994,  and  (2)  the  Man¬ 
aged  Era,  or  post- 1994  period. 

This  chapter  also  introduces  the  Department’s  M&S  management  framework  for  es¬ 
tablishing  credible  simulations,  including  classes  of  simulation,  simulation  hierarchies,  ini¬ 
tiatives  to  establish  distributed  simulation  capabilities,  the  expanding  need  of  credibility  by 
the  major  stakeholders  influencing  the  Department’s  M&S  management  priorities:  acquisi¬ 
tion,  training,  operations,  analysis,  and  experimental  domains.  Research  indicated  that 
M&S  credibility  was  a  multi-dimensioned  problem.  Chapter  III  through  Chapter  VII  re¬ 
view  different  dimensions  of  the  Department’s  M&S  credibility  challenge. 

B.  LITERATURE  SEARCH 

A  significant  body  of  literature  evolved  since  the  mid-1970s  focused  on  the  need  for 
major  systemic  improvements  in  M&S  software  credibility  as  a  basis  for  supporting  higher 
levels  of  confidence  of  simulation  results  by  Department-  and  National-level  decision¬ 
makers.  The  taxonomy  shown  in  Figure  2-1  represents  a  synthesis  of  the  major  research 
areas  for  models,  simulations,  verification,  validation,  and  accreditation  affecting  the  credi¬ 
bility  of  simulations  and  Department-  or  National-  level  decision-makers  confidence  in  the 
Department’s  simulation  or  federation  results. 

Unfortunately,  there  is  no  “silver  bullet”  for  improving  simulation  software  credi¬ 
bility.  Research  indicates  that  improving  the  credibility  of  the  Department’s  computer- 
based  M&S  software  supporting  enhanced  user  confidence  involves  the  following  multiple 
interrelated  domains,  competencies,  disciplines,  and  bodies  of  knowledge  illustrated  in 


Figure  2-1  and  discussed  in  the  following  chapters: 

•  Department  of  Defense  M&S  Credibility  Measures - Chapter  III 

•  Heterogeneous  System  Representation  Anomalies - Chapter  IV 

•  The  Architectural  Framework - Chapter  V 
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•  Simulation  Software  Quality  Attributes - Chapter  VI 

•  Process  Improvement  and  Software  Engineering - Chapter  VII 

•  Software  Product  Lines  and  Architecture - Chapter  VIII 

•  Domain  Analysis - Chapter  IX 


Figure  2- 1 .  Major  Research  Areas  Relevant  to  this  Work 


M&S  credibility  has  also  taken  on  additional  importance  and  emphasis  as  the 
Department  employs  large-scale,  legacy  M&S  supporting  Department  missions  and 
national  security  initiatives.  The  literature  survey  found  the  use  of  M&S  software 
pervasive  in  many  domains,  expanding  rapidly  into  many  new  areas,  and  experiencing  a 
fast-growing  demand  for  even  more  credible  applications  and  uses: 

•  United  States  Homeland  security  and  national  security  strategy  IBus02a,  Bus02b, 
Bus02c,  Bus02d]; 

•  Department  policy,  memoranda,  and  strategy  documents  [Kri88,  Car88,  DOT89, 
Kri89,  Pai97,  Gan98,  Gan99,  DoDOOb,  GMWOOa,  GMWOOb,  MonOOa,  MonOOb, 
SheOOa,  HRA+01,  WolOl,  RumOl,  Rum02a,  Rum02b,  Ald02a,  Ald02b,  Ald02c, 
Ald02d,  DoD02d,  DoD03a,  DoD03b,  D0D03c]; 
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•  United  States  Government  Accounting  Office  (GAO)  Reports  [PAD78,  PAD79a, 
PAD79b,  PAD80,  Mye82a,  Mye82b,  She82,  Con86a,  Con86b,  Che87,  Bre88, 
Bro88,  Bro91,  Dav91,  HSW+92,  Con93,  Geb93,  Geb94a,  Geb94b,  Geb95,  Dav96, 
ZHF98,  Bro99,  HDK+OO,  HEE+OO,  McCOO,  RFH+OO,  BCP+Ol]; 

•  Defense  and  Army  Science  Board  studies  [WHM+79,  JBC+88,  BTE+93,  HW96, 
GKB+97,  LWF+98,  HWB+98,  MCC+98,  BBB+99,  FEE+99,  FGW+99,  GBB+99, 
GWB+99,  HAC+99,  LLD+99,  MSI+99,  HNC+OO,  MorOOb,  BCC+Ola,  FOP+Ol, 
VFH+01]; 


•  Private  and  Government  sector  studies,  analyses,  and  reports  [Qua64,  SCC+88, 
DOT89,  DB91,  LA92,  Dow92,  DH92a,  DH92b,  ABD+93,  CSC94a,  CSC94b, 
Coh94,  SG94,  Bec94,  EBG+94,  Par94,  PBA+94,  PMR94,  JAS95,  MCU+95, 
HBM96,  PHP+96,  Por96,  WSM+96,  CCK+96,  JAS96,  JAS97a,  JAS97b,  JAS97c, 
Lie97,  Gau98,  NMY+98,  SG99,  BCE+99,  DOTOO,  SBAOO,  BFOO,  COHOO, 
MWD+00,  DMSOOb,  RGHOO,  CraOla,  CraOlb,  DDH01,  DOTOla,  DOTOlb, 
HarOla,  HarOlb,  HCG+Ola,  HCG+Olb,  NelOl,  SBAOla,  SB  AO  lb,  RTI02];  and 


•  Department  Inspector  General  reports  [RSV+93,  BSV+97,  GGU+97,  AMD98, 
GUS+00]. 

M&S  software  credibility  areas  that  affect  decision-makers’  confidence  in  simula¬ 
tion  results  and  authoritative  model  representations: 


•  Software  engineering  and  development  processes  based  on  best  practices  [You89, 
BL91,  You93,  Coh94,  Dem95a,  Som95,  HCP96,  Hoe96,  Wie96b,  Pre97,  HBL97, 
Hin97,  CS97,  BW97,  Rub97,  RFL+98,  Roa98,  Woo98,  HHK+99,  BasOO,  RSF+00, 
BDA+00,  HHPOO,  HNC+OO,  RFH+OO,  BCC+Olb,  KMOla,  RakOl,  CHS+02, 
MR02,  Li02,  LQZ02]; 

•  Process  improvement  and  reengineering  [Sch9 1 ,  HC93 ,  Hal95 ,  HamO  1  b, 
PWC+95,  FC99,  MCR+00,  BBF+01]; 

•  Modeling  and  simulation  improvement  [BKR95,  Ham96,  HNP97,  Hug97a,  CR98, 
Koo99,  HamO  la,  LKOO,  BCB02,  Bee02];  and 

•  Emerging  bodies  of  knowledge  (BOK)  in  related  knowledge  areas  such  as  soft¬ 
ware  quality  measurement  [HM96,  Sch02a],  project  management  [Fra94,  Dun96], 
and  modeling  and  M&S  BOK  [ACP+99],  the  Department’s  joint  technical  architec¬ 
tures  [JTA02a,  JTA02b],  sound  knowledge  engineering  practices  for  conceptual 
model  development  [Pac99a,  PacOla]  and  verification  and  validation  [IEE86, 
IEE98b]  of  the  conceptual  model,  data,  and  simulations  [Pac02a,  Bor02]  conducted 
during  the  entire  M&S  life-cycle. 
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A  top-level  literature  survey  was  also  conducted  for: 

•  Studies  in  the  Department’s  Missile  Defense  M&S  domain  [Bra96,  Li99,  ISE99, 
WelOO,  BraOOa,  BraOOb], 

•  Performance  recommendations  supporting  improved  information  technology  in¬ 
vestments  within  the  Department  and  federal  government  [SBD+95,  BPC+96, 
MLW+96,  AVL+97,  Hin97,  Pai97,  Dod98,  PKM+98,  Bro99,  MccOO,  RodOO, 
RSF+00,  CLP+01,  CHS+02,  FNP+02,  BBF02]. 

•  The  scope  and  magnitude  of  simulation  software  credibility.  The  problem  with 
simulation  credibility  appears  to  be  a  systemic  issue  extending  well  beyond  the  De¬ 
partment’s  purview,  and  found  in  different  domains  and  diverse,  non-Department 
systems  such  as  economic  models  [PosOO,  KBA+02],  transportation  [AndOO],  en¬ 
ergy  [Sta74],  supply  [PSA79],  environmental  protection  [DMF+95],  retirement 
forecasting  [Dod97,  Che86a,  Che86b],  supercomputing  [Bro91],  air  pollution 
[GMW89,  DMP+97],  credit  subsidies  [Cal97],  highway  management  [SchOla],  nu¬ 
clear  weapons  [CFM99,  HEC91],  and  manufacturing  [AROO]. 

Finally,  the  Department  M&S  software  programs  must  support  and  comply  with 
applicable  Federal  Government  and  Department  requirements,  software  development  proc¬ 
esses  and  product  security  measures  including: 


•  Information  operations  /  assurance  and  critical  infrastructure  protection  re¬ 
quirements  [EFF+97,  FX99,  ABBOO,  BusOl,  FOP+Ol,  GPH+01,  WFP01,  AFF02], 

•  Interoperability  standards  [HKF96,  HMD99,  MCF+99,  HMOO,  BFB+01, 
BLR+02],  and 

•  Legal  requirements  for  information  technology  and  performance  measurements 
[CC96,  ATF01,  CosOl,  GlaOl,  Man02,  AS02a,  AS02b]. 

C.  THE  DEPARTMENT’S  MODEL  AND  SIMULATION  FRAMEWORK 

1.  Background 

National-  and  Department-level  decision-makers  need  high-quality  computer-based 
simulation  models  and  modeling  frameworks,  supported  by  valid  qualified  and  quantifiable 
levels  of  process  and  product  credibility.  The  quest  for  confidence  covers  an  ever- 
expanding  range  of  M&S  ranging  from  high-fidelity  engineering  models  to  highly  aggre¬ 
gated  campaign-level  M&S.  The  potential  spectrum  of  Department  uses  for  M&S  grows 
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continuously  when  one  considers  that  “everything  is  a  simulation  except  com¬ 
bat, ’’[BTE+93],  The  terms  model  3  and  simulation54  have  several  definitions,  including 
Law  and  Kelton  [LKOO]55,  the  IEEE  [IEE90]56,  and  the  Department  [DoD94].  In  this  re¬ 
search  we  use  the  Department’s  definitions:  a  model  “is  a  physical,  mathematical,  or  oth¬ 
erwise  logical  representation  of  a  system,  entity57,  phenomenon,  or  process”  [DoD94], 
while  a  simulation  is  “a  method  for  implementing  a  model  over  time”  [DoD94]. 

John  von  Neumann  first  proposed  the  concept  of  modern  digital  simulation  as  the 
application  of  gathering  repetitive,  statistical  data  on  modeled  phenomena,  via  the  Monte 
Carlo  method  [Rot83].  [Tho97a]  provides  an  historical  account  of  computer-based  simula¬ 
tion  from  the  1940s  wartime  combat  operations  research  role  and  focus  on  improving  the 
use  of  weapons,  through  the  1960s  use  of  M&S  for  systems  analysis.  From  1940  to  1997, 
there  have  been  four  major  eras  in  the  evolution  of  software  during  the  computer  eras  af¬ 
fecting  the  development  of  Department  M&S  software  [Pre97]: 

•  The  early  years,  characterized  by  custom  software,  batch  operations  and  limited 
software  distribution, 

•  The  second  era,  from  the  mid-1960s  to  the  late  1970s,  experienced  software  prod¬ 
ucts  developed  by  the  private  sector,  multi-user  computer  systems,  on-line  data¬ 
bases,  and  the  first  real-time  systems,  and  the  need  for  software  maintenance  grew, 


A  model  is  a  mathematical  representation  of  a  system,  where  a  system  is  any  sort  of  process  or  structure.  A  process  model  normally 
operates  sequentially  over  time,  but  may  use  other  dimensions.  Sequential  processes  in  time  are  modeled  as  an  independent  variable  over 
a  linearly  ordered  set  using  real  numbers  (continuous  time)  or  integers  (discrete  time).  A  model  has  two  types  of  system  parameters,  or 
variables  characterized  by  range  (the  set  from  which  it  assumes  a  value)  and  type  (random  or  deterministic).  A  deterministic  variable 
assumes  a  unique  from  the  range  at  each  point  in  time,  while  a  random  variable  is  described  statistically  probability  distribution  over  the 
range.  System  variables  may  be  for  input  or  output,  and  the  resulting  model  may  be  static  or  dynamic.  In  a  static  model,  the  outputs  at 
any  time  are  only  a  function  of  the  inputs  at  that  time,  possess  no  memory  and  has  no  recollection  of  previous  events,  whereas  the  outputs 
of  a  dynamic  system,  may  depend  upon  past  values,  as  well  as  present  inputs,  thus  the  system  remembers  certain  information  about  pre¬ 
vious  inputs.  The  remembered  information  is  called  the  state  of  the  system.  A  structural  model  usually  involves  the  representation  of 
relations  e.g.,  airline  flight  route  [Mar83a] 

54 

A  simulation  is  the  representation  of  certain  features  of  the  behavior  of  a  physical  or  abstract  system  by  the  behavior  of  another  sys¬ 
tem.  Simulations  employ  computerized  models  of  certain  significant  features  of  some  physical  or  logical  system.  The  object  of  the  simu¬ 
lation  process  is  to  provide  an  experimental  model  for  the  accumulation  of  data  on  the  target  system,  and  is  comprised  of  the  following 
steps:  experiment  definition,  modeling,  computer  implementation,  validation,  and  data  gathering  [Rot83]. 

The  technique  to  imitate  or  simulate  the  real-world  operations  of  facilities,  processes,  or  systems  supporting  scientific  study;  which 
require  a  set  of  assumptions  about  how  it  works.  The  assumptions,  usually  mathematical  or  logical  relationships  constitute  a  model,  used 
to  gain  an  understanding  of  how  the  corresponding  study  behaves.  A  simulation  employs  a  computer  to  evaluate  a  model  numerically, 
collect  data  to  estimate  the  desired  true  characteristics  of  the  model  [LKOO]. 

The  IEEE  defined  a  simulation  as  “(1)  a  model  that  behaves  or  operates  like  a  given  system  when  given  a  set  of  controlled  inputs;  (2) 

the  process  of  developing  or  using  a  a  model  as  in  (1)  [IEE90]. 

57  ....  . 

An  entity  is  a  distinguishable  person,  place,  unit,  thing,  or  concept  about  which  information  is  kept  [DoD98]. 


23 


•  The  third  era,  from  the  mid-1970s  lasted  over  a  decade  and  ushered  in  a  period  of 
low-cost  computation  power  for  consumer  use,  saw  the  development  of  distributed 
systems,  and  witnessed  the  initial  efforts  into  artificial  intelligence  development, 

•  While  the  fourth  era,  from  the  mid-1990s  to  the  present,  included  mobile  and  dis¬ 
tributed  systems,  the  Internet,  network/parallel  computing,  expert  systems,  neural 
networks,  powerful  desktop  personal  computers,  object-oriented  technologies,  and 
the  software  to  integrate  legacy  systems  [Pre97]. 

During  these  four  periods  the  M&S  community  experienced  the  development  of  new  com¬ 
puter-based  models  and  simulations,  a  growing  concern  with  credibility,  emerging  trends  of 
repeated  use  of  models  for  different  applications  and  the  development  of  families  of  models 
or  hierarchies. 

2.  Classes  of  Simulations 

The  current  Department  M&S  framework  has  three  M&S  classes:  live,  virtual,  and 
constructive,  supporting  five  major  customer  domains:  training,  operations,  analysis,  ex¬ 
perimentation  and  acquisition  [DoD02d].  Live  simulation  refers  to  real  people  operating 
real  systems  including  field  training  exercises  involving  troops  and  actual  equipment, 
which  allow  soldiers  to  use  organizational  equipment  under  actual  environmental  condi¬ 
tions,  simulating  combat.  The  live  simulation  also  provides  ground  test  data  on  actual 
hardware  and  software  performance  in  an  operational  or  development  environment.  Live 
test  data  supports  simulation  validation  (e.g.,  calibration)  using  the  model-test-model  con¬ 
cept  employed  in  the  Department’s  acquisition  process. 

Virtual  M&S  involves  real  people  operating  simulated  systems,  digital  representa¬ 
tions  of  environments,  organizations,  systems,  other  entities,  and  processes.  Virtual  M&S 
put  the  human  in  the  loop  in  a  central  role  by  exercising  motor  control  skills,  decision 
skills,  or  communication  skills.  Current  state-of-the-art  virtual  M&S  bring  the  system  or 
subsystem  and  its  operators  together  in  a  synthetic  or  simulated  environment.  Virtual  M&S 
run  in  real  time  immersing  players  in  synthetic  environments  or  simulators.  Virtual  M&S 
may  include  actual  hardware,  stimulated  by  the  output  of  computer  simulations. 
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Constructive  M&S  involves  simulated  people  operating  simulated  systems.  Con¬ 
structive  simulations  employ  digital  representations  of  environments,  organizations,  sys¬ 
tems,  other  entities,  and  processes,  operated  with  human  players,  or  without  human  players. 
Human  player  interactions  are  limited  to  controlling  units,  and  generally  referred  to  as 
wargames.  Human-in-the-loop  (HIL)  simulations  provide  the  stress  and  decision-making 
associated  with  live  simulations,  and  permit  the  introduction  of  multiple  types  of  platforms 
for  evaluation  of  actual  interaction  and  interoperability. 

Categories  of  constructive  M&S  consist  of  computer  models,  wargames,  and  ana¬ 
lytical  tools  used  across  a  range  of  activities.  These  activities  include  detailed  engineering 
design  [Law02]  and  cost  [KanOl]  or  subsystem  and  system  performance  calculations  to 
support  development  of  technical  specifications.  Higher-level  constructive  M&S  provides 
information  on  the  outcomes  of  battles  or  major  campaigns  involving  joint  or  combined 
forces;  identify  mission  needs;  and  support  operational  effectiveness  analyses  measures  of 
effectiveness  (MOE),  loss  exchange  ratios  (LER),  force  exchange  ratios  (FER),  and  meas¬ 
ures  of  performance  (MOP).  Stochastic58  or  deterministic59  simulations  approximate  the 
real  world.  Other  uses  of  constructive  M&S  include  developing  life  cycle  cost  estimates, 
supportability  analyses,  and  force  management  analysis  of  alternatives. 

3.  Simulation  Model  Hierarchies 

The  concept  of  a  hierarchy  of  simulation  models  evolved  in  Europe  and  the  United 
States  during  the  1970s  with  the  objective  of  managing  M&S  complexity60.  However,  the 
various  hierarchy  models  lacked  integration  and  usually  consisted  of  a  high-level  model 
and  several  lower  level  M&S.  [Ham96,  HNP97]  suggest  that  hierarchical  modeling  sup¬ 
ports  abstraction61.  A  variable  resolution  M&S  concept  for  integrating  heterogeneous 
M&S  into  the  hierarchy  has  existed  since  the  early  1980s  [DH92a,  DH92b]. 


58  The  theory  of  stochastic  processes  deals  with  events  that  develop  in  time  or  space  and  which  cannot  be  described  precisely  except  in 

terms  of  probability  theory  [Mar83b]. 

59  ...  . 

A  deterministic  variable  assumes  a  unique  value  from  the  range  of  set  values  at  each  point  in  time  [Mar83a]. 

Complexity  refers  to  time  and  space  resources  needed  to  simulate  a  model,  and  is  a  dimension  independent  from  resolution,  although 
they  are  often  related  in  a  monotonic  manner  [Zei92b] 

Abstraction  isolates  important  aspects  of  the  problem  and  suppresses  the  unimportant  aspects  [HNP97]. 
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[HHPOO]  suggests  the  literature  includes  many  other  different  M&S  taxonomies. 
For  example,  [Zei92a]  addressed  model  representations  in  modular  hierarchies,  while 
[Sch92]  detailed  model  hierarchies  of  different  resolutions.  Providing  other  viewpoints, 
[HBF92]  focused  on  maintaining  process  independence  between  different  levels  of  resolu¬ 
tion;  while  [BD02]  identified  hierarchical  model  framework  to  support  improved  design 
quality  of  object-oriented  systems;  whereas  [Rei92]  adopted  a  different  five-level  capabili- 
ties  assessment  viewpoint  .  Lacking  an  architectural  foundation,  many  of  these  hierarchy 
models  depended  on  broad  categorical  definitions,  heuristics,  subject  matter  experts,  and 
general  qualifications  to  match  an  existing  M&S  to  the  model  framework. 

[Hug97a,  Hug97b]  proposes  there  is  not  a  universally  accepted  way  to  categorize 
M&S,  since  M&S  range  over  a  vast  spectrum  of  fidelity,  precision,  and  resolution  from  de¬ 
tailed  engineering  representations  to  highly  aggregated  representations  of  force-on-force 
engagements,  often  involving  multiple  dimensions  of  structural,  functional,  temporal,  and 
qualitative  abstraction,  which  are  not  complete  or  necessarily  independent  of  each  other. 
[Ham96,  HNP97]  cite  the  close  relationships  between  the  concepts  of  aggregation  and  de¬ 
composition  with  the  levels  of  abstraction  concept.  [Ham96,  HNP97]  also  note  that  ab¬ 
straction  (e.g.,  ignoring  behavior  or  minor  differences)  indicates  a  simplification  strategy, 
whereas  abstraction  by  lumping  together  indicates  the  use  of  an  aggregation  method. 
[Ham96,  HNP97]  further  suggest  the  use  of  an  open  architecture  as  a  means  of  developing 
simulations  with  the  flexibility  to  support  multiple  levels  of  aggregation. 

With  the  Department’s  growing  need  to  address  human  behavior,  [MT01]  proposed 
a  hierarchical  model  composed  of  virtual  individuals,  groups,  and  crowds.  [Sta97]  pro¬ 
vides  a  different  perspective,  suggesting  that  a  hierarchy  of  linked  measures  of  merit—  in¬ 
cluding  measures  of  perfonnance,  measures  of  functional  performance,  and  measures  of 
force  effectiveness—  may  improve  acquisition  activities.  Finally,  [PMR94]  introduced  a 
Notional  Hierarchy  of  Technologies63  with  five  categories  for  assigning  M&S  enabling 


62  The  capabilities  assessment  levels  built  on  weapons  system  level  analysis,  and  added  a  supporting  analyses  level,  program  alternative 
analyses,  mission  analyses,  and  at  the  top  of  the  pyramid,  campaign  analysis  [Rei92]. 

The  enabling  technologies  included  standards  and  protocols  at  he  lowest  level,  and  added  layers  for  underlying  collaborative  tech¬ 
nologies,  utilities  and  infrastructure,  applications,  and  an  integrated  acquisition  environment  at  the  top  [PMR94]. 
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technologies:  standards  and  protocols,  collaborative  technologies,  utilities  and  infrastruc¬ 
ture,  applications,  and  at  the  top,  an  integrated  acquisition  environment. 

The  Department’s  simulation  hierarchy  includes  four  categories  of  M&S  for  com¬ 
municating  the  scope  or  level  of  model  detail:  engineering,  engagement,  mission  and  cam¬ 
paign,  [PMR94,  CS97],  generally  with  more  abstraction  moving  up  the  simulation  hierar¬ 
chy  and  more  refinement,  or  level  of  detail  moving  down  the  simulation  hierarchy.  The 
M&S  characterized  in  the  hierarchy  represents  increasing  aggregation  from  the  engineering 
models  at  the  base  of  the  hierarchy  to  the  highly  aggregated  representations  supporting  a 
theater  level  view  [HM95].  Not  every  model  or  simulation  fits  perfectly  into  one  of  the 
classifications,  and  M&S  fidelity  quality,  addressed  later,  can  vary  at  any  point  in  the  M&S 
hierarchy. 

The  engineering  M&S  include  very  detailed  simulations,  concentrating  on  individ¬ 
ual  components  and  their  interaction,  with  application  in  design  analysis,  risk  mitigation  in 
component  performance  and  tradeoff,  requirements  specification,  and  perfonnance  analy¬ 
sis.  Engagement  M&S  usually  depict  friendly  versus  enemy  engagements,  and  generally 
provide  some  type  of  system  effectiveness  outcome  supporting  analysis  of  alternatives 
(AoA),  requirements  evaluation,  system  effectiveness,  tactics,  techniques,  and  procedures 
(TTP)  development  and  assessments,  tradeoff  analysis,  and  test  support.  Virtual  prototypes 
usually  run  as  a  simulation  at  the  engagement  level  and  above. 

Mission/Battle  level  M&S  depict  multi-platform,  multi-task  force  packages  and 
usually  provide  some  type  of  mission  effectiveness  outcome.  Mission/Battle  M&S  support 
AoA  and  requirements  evaluation,  deployment  analysis,  weapons  integration,  TTP  devel¬ 
opment  and  assessments,  training,  and  wargaming.  At  the  apex  of  the  M&S  hierarchy  sit 
the  highly  aggregated  Theater/Campaign  models.  These  M&S  also  support  AoA,  require¬ 
ments  evaluation,  TTP  development  and  assessment,  war  gaming,  and  battle-staff  training. 
Figure  2-2  is  an  annotated  illustration  of  the  current  Department  M&S  hierarchy  [PMR94]. 
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Figure  2-2.  Department  M&S  Hierarchy  (After  [PMR97]) 


4.  Distributed  Simulations 

The  Department  supports  three  architectural  implementations  for  distributed  M&S: 
the  Aggregate-Level  Simulation  Protocol  (ALSP),  the  Distributed  Interactive  Simulations 
(DIS)64  application  protocols  [IEE95,  IEE98a],  and  the  most  recently  approved  method,  the 
HLA.  For  distributed  simulations  [Ham96,  HNP97],  an  individual  simulation  called  a  fed¬ 
erate  interoperates  with  other  selected  federates  through  a  common  technical  framework, 
such  as  the  HLA,  forming  federations. 

The  IEEE  supports  the  HLA  rules,  IEEE  1516/DI  [IEE98e],  HLA  object  model 
template  specification  [IEEOOc],  HLA  federate  interface  specification  [IEEOOd],  and  HLA 
object-model  template  [IEEOOe].  The  HLA  object-model-template  specification  [IEEOOe] 
defines  the  format  and  syntax  of  HLA  object  models,  but  not  the  content.  With  the  evolu¬ 
tion  of  the  HLA  beyond  the  Department,  a  growing  demand  for  interoperability,  and  ap- 

64 

DIS  is  a  synthetic  environment  within  which  humans  may  interact  through  simulations  at  multiple  sites  networked  sing  compliant 
architecture,  modeling,  protocols,  standards,  and  databases  [DoD95]. 
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proval  of  the  HLA  standard  by  the  IEEE,  the  definitions  employed  in  this  research  will  re¬ 
main  consistent  with  approved  IEEE  definitions  (e.g.,  accuracy65,  resolution66)  whenever 
possible.  Legacy  Department  definitions  included  in  the  research  provide  context  for  deci¬ 
sions,  issues,  and  possible  ambiguities  faced  by  Department  simulation  developers.  Inter¬ 
estingly  enough,  the  tenn  fidelity,  previously  defined  by  the  Department  [DoD98],  and  used 
liberally  in  the  M&S  literature,  is  not  included  in  the  IEEE’s  standard  glossary  of  software 
engineering  terms  [IEE90],  or  the  HLA  object  model  template  specification  [IEEOOe]. 

5.  The  Growing  Need  for  Credible  Models  and  Simulations 

As  M&S  technology  developed  in  the  Department,  several  functional  areas,  includ¬ 
ing  the  acquisition,  training,  operations,  analysis,  and  experimental  functional  domains  in¬ 
corporated  simulation  technology  into  their  infrastructure  with  the  goal  of  enhancing  per¬ 
formance,  reducing  overall  costs,  and  attaining  improved  schedule  milestones.  In  all  of 
these  functional  domains,  M&S  is  now  a  key  component  of  the  domain  toolbox,  although 
the  past  performance  of  simulation  technology  has  not  always  met  expectations. 

a.  The  Department’s  Acquisition  Domain 

The  Department’s  acquisition  domain,  including  developmental  and  opera¬ 
tional  test  and  evaluation  communities,  employ  M&S  in  the  acquisition  of  new  operational 
systems  [DoD03b,  DoD03c].  The  Department  also  incorporated  M&S  into  the  acquisition 
process  to  reduce  cost,  improve  performance,  maintain  schedule  and  mitigate  risk.  The  re¬ 
quirement  for  credible  M&S  results  are  incorporated  in  the  Department  acquisition  direc¬ 
tive  [DoD03b]  and  instructions  [DoD03c],  which  state, 

•  The  conduct  of  test  and  evaluation,  integrated  with  modeling  and  simula¬ 
tion,  shall  facilitate  learning,  assess  technology  maturity  and  interoperabil¬ 
ity,  facilitate  integration  into  fielded  forces,  and  confirm  perfonnance 
against  documented  capability  needs  and  adversary  capabilities  as  described 
in  the  system  threat  assessment  [DoD03b]. 


65  Accuracy  according  to  the  IEEE  is  the  measure  of  the  maximum  deviation  of  an  attribute  or  parameter  value  in  the  simulation  or  fed¬ 
eration  from  reality  or  some  other  chosen  standard  or  referent  [IEEOOe]. 

The  IEEE  defines  resolution  as  the  smallest  resolvable  value  separating  attribute  or  parameter  values  that  can  be  discriminated.  Reso¬ 
lution  may  vary  with  magnitude  for  certain  data  types  [IEEOOe]. 
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•  The  T&E  [test  and  evaluation]  strategy  shall  provide  information  about  risk 
and  risk  mitigation,  provide  empirical  data  to  validate  model  [DoD03c] 

•  Adequate  time  and  resources  shall  be  planned  to  support  pre-test  predictions 
and  post-test  reconciliation  of  models  and  test  results,  for  all  major  tests. 

[DoD03c], 

In  addition,  “The  PM  shall  pan  for  M&S  resources  throughout  the  acquisition  life 
cycle.  The  PM  shall  identify  and  fund  M&S  resources  early  in  the  life  cycle” 

[DoD03c], 

In  the  area  of  acquisition  reform  [WHA+00],  the  National  Perfonnance  Re¬ 
view  (NPR)  established  a  Department  acquisition  reform  objective  of  reducing  new  system 
delivery  time  by  twenty-five  percent,  subsequently  increased  by  the  Department  to  fifty 
percent  [ACP+99],  in  part  by  acquisition  refonn  initiatives  capitalizing  on  engineering 
trade-offs  and  M&S  performance  modeling  to  achieve  the  NPR  goals.  Simulation  software 
also  supports  the  evolving  Department  transformation67  objectives  [OAH+97,  DGH+98, 
RumOl,  Rum02a,  Rum02b]  to  improve  the  quality,  interoperability,  effectiveness,  cost, 
performance,  and  schedule  metrics  of  the  Department’s  business  and  warfighting  systems 
supporting  the  National  Military  Strategy  [HBM96,  Rum02b]. 

In  the  acquisition  arena,  transfonnation  includes  the  replacement  of  inflexi¬ 
ble  requirements-based  methods  with  a  capabilities-based  methodology  [RumOl]  supported 
by  evolutionary  acquisition  practices  [Ald02d]  for  major  Department  acquisitions  including 
the  Nation’s  missile  defense  program.  The  success  of  M&S  to  support  the  Department’s 
transfonnation  process  hinges  in  part,  on  the  acceptance  and  successful  implementation  of 
new  or  enhanced  M&S  methods  and  techniques  such  as  the  Simulation,  Test,  and  Evalua¬ 
tion  Process  (STEP)  [CS97],  Simulation-Based  Acquisition  (SBA)  [San97b,  San97c,  FT98, 
JMS98,  San98b,  SBB+98,  Bro99,  GS99,  San99,  SBA99,  SBB+99a,  SBB+99b,  ATOO, 
SBAOO,  BGK+00,  KLM+00,  KMOlb,  ZitOl,  ECP02],  Distributed  Product  Descriptions 
(DPD)  [HHOOb],  the  Army’s  Simulation  &  Modeling  for  Acquisition,  Requirements,  and 
Training  (SMART)  program  [BiaOO,  EKHOO,  EKH01],  and  the  development  of  new  col¬ 
laborative  environments  [San98a,  San98b,  San99,  HolOl]  including  the  Joint  Synthetic  Test 

67  Transformation  results  from  the  exploitation  of  new  approaches  to  operational  concepts  and  capabilities,  the  use  of  old  and  new  tech¬ 
nologies,  and  new  forms  of  organization  that  more  effectively  anticipate  new  or  still  emerging  strategic  and  operational  challenges  and 
opportunities  and  that  render  previous  methods  of  conducting  war  obsolete  or  subordinate  [RumOl] 

30 


and  Evaluation  Battlespace  [San98b,  Bra98,  HolOl],  the  Advanced  Distributed  Simulation 
(ADS)  [Kec99]  capability,  and  testing  system-of-systems  interoperability  (e.g.,  the  Single 
Integrated  Air  Picture)  [Dah02]. 

The  Revolution  in  Business  Affairs,  a  Department  acquisition  reform  im¬ 
perative,  leverages  M&S  to  reduce  Total  Ownership  Cost  and  improve  delivery  for  new 
systems  by  employing  SBA  processes  [Eva96,  Lav97,  San97b,  BSB+98,  HolOl]  among 
other  best  business  practices.  The  vision  for  the  Department’s  SBA  initiative  is  “an  acqui¬ 
sition  process  in  which  the  Department  and  industry  are  enabled  by  robust,  collaborative 
use  of  M&S  technology  integrated  across  acquisition  phases  and  programs”  [San97b].  The 
objective  of  SBA  is  to  reduce  schedule  time  and  risk  in  the  development  and  maintenance 
of  a  product  with  an  improved  quality  at  a  lower  cost  in  a  collaborative  government/private 
sector  environment  employing  an  Integrated  Product  and  Process  Development  methodol¬ 
ogy  over  the  system  life  cycle.  However,  Department  support  for  these  initiatives  requires 
the  development  of  business-case  framework  to  evaluate  the  investment  opportunity  and 
the  potential  return-on-investment  [BGK+00]. 

STEP  is  an  iterative  model-test-model  approach  with  a  goal  of  improved 
M&S  credibility,  test  community  confidence,  and  a  better  understanding  of  the  strengths 
and  limitations  of  complex  models,  M&S  and  data  by  decision-makers.  The  STEP  para¬ 
digm  of  model-simulate-fix-test-iterate  [CS97]  embodied  in  the  DoD  developmental  and 
operational  test  and  evaluation  (T&E)  process  has  the  potential  to  support  improved  confi¬ 
dence  in  tests  and  evaluation  results. 

Department  testers  [DOT89,  Bra98,  Kec99,  BCE+99,  Pau99,  Obr99, 
CoyOOa,  CoyOOb,  DOTOla,  MOR02]  also  seek  answers  to  where  M&S  may  provide  credi- 
ble  information  and  under  what  conditions  M&S  can  replace  testing.  Hydrocode  models 
track  shock  propagations  and  materiel  distortion  to  support  vulnerability/lethality  assess¬ 
ments  [TIL+95,  MH98]  and  support  Department  Live  Fire  Test  &  Evaluations  (LFT&E). 
The  LFT&E  testers  characterize  the  results  of  fragment  and  kill-vehicle  interaction  with  the 
target,  which  are  subsequently  incorporated  into  lethality  M&S.  Current  research  into  hy- 


68  Hydrocodes  are  physics-based,  first  principle  M&S  of  hyper  velocity  solids  interaction  used  in  vulnerability  and  lethality  M&S  to 
track  shock  propagation  and  distortions  of  multiple  materials  [ID  AO  1  ] 
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drocodes  models  cited  by  [IDA01]  indicates  current  hydrocode  model  technology  is  limited 
to  the  primary  effects  of  a  weapon  on  a  system  and  has  major  voids  in  secondary  weapons 
effects,  suggesting  that  M&S  can  complement,  but  not  replace  live-fire  tests. 

[DOTOlb]  cites  the  quality  of  Department  software  testing  as  an  ongoing 
systemic  issue,  needing  major  improvements  in  the  area  of  M&S  support  for  improved  test 
and  evaluation  readiness  capabilities.  [CoyOOa]  supports  this  assessment  and  suggests  ad¬ 
ditional  emphasis  in  the  area  of  best  software  test  processes  to  improve  the  quality  of  the 
Department’s  operational  testing  program.  However,  [BCE+99]  notes  that  simulating 
complex  system  interactions,  limited  by  cost,  time,  personnel,  and  technology,  adversely 
impacts  the  accuracy  and  fidelity  of  the  M&S  supporting  Departmental  test  programs. 

A  Departmental  M&S  survey  [DDH01,  HBD+01]  sponsored  by  the  Direc¬ 
tor,  Operational  Test  and  Evaluation  (OT&E)  to  gain  insight  and  confidence,  performed  a 
limited  review  of  Department  M&S  including  types,  characterization,  management,  man¬ 
agement  activities,  cost,  and  noted  for  VV&A  that  much  policy  guidance  has  been  devel¬ 
oped,  but  implementation  appears  minimal.  The  increasing  dependence  on  M&S  noted  by 
[DDH01]  added  increased  emphasis  on  management  activities  considered  critical  to  suc¬ 
cess  including: 

•  M&S  Support  Plan 

•  M&S  Staff 

•  VV&A  Plan  /  processes 

•  M&S  reuse 

•  Collaborative  environment 

•  Performance  incentives  [DDH01,  HBD+01]. 

The  February  2001  Department  M&S  Workshop  for  Assuring  M&S  Credi¬ 
bility  for  Defense  Acquisition  and  T&E:  Survivability,  Lethality  and  Mission  Effectiveness 
[DOTOla]  corroborated  the  results  from  [DDH01,  HBD+01].  The  Workshop  identified 
several  systemic,  ongoing  factors  hampering  M&S  and  VV&A  programs,  and  concluded 
that  acquisition  program  managers  need  more  help  implementing  M&S  and  VV&A  within 
funding  constraints  [DOTOla].  Furthermore,  [DOTOla]  cautioned  there  is  “no  definitive  or 
reliable  link  between  following  M&S  and  VV&A  policy  and  technical  guidance  and  having 
confidence  in  M&S  credibility  for  acquisition  applications”  [DOTOla],  As  a  result  of  these 
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systemic  issues,  [DoDOla]  concluded  that  the  lack  “of  adequate  VV&A  resulted  in  uncer¬ 
tain  or  poor  representations  of  military  forces  in  M&S,  thereby  causing  negative  training, 
inaccurate  analyses,  and  lack  of  trust  in  M&S’’  [DoDOla]. 

An  October  2002  Military  Operations  Research  Society  (MORS)  M&S 
Workshop  [GF02]  addressed  relevant  topics  to  this  study  including:  the  status  and  health  of 
the  Department’s  M&S  policy,  the  capability  versus  expectations  for  using  M&S  in  De¬ 
partmental  acquisition,  the  Department’s  VV&A  practices,  and  addressed  the  state  of  the 
Department’s  M&S  credibility  since  [DDH01,  HBD+01,  and  DOTOla].  The  workshop 
members  concluded  that  the  Departmental  M&S  policies  were  adequate  and  identified  sev¬ 
eral  areas  needing  additional  emphasis  including  risk  mitigation,  operational  assessments, 
and  test-prediction  and  post-test  reconciliation  [GF02], 

The  workshop  members  also  noted  that  most  of  the  Department’s  M&S  ef¬ 
fort  appeared  in  areas  requiring  the  highest  fidelity,  although  current  Departmental  M&S 
policies  or  funding  levels  do  not  support  these  efforts.  Specific  areas  of  concern  included: 
evaluations  beyond  the  bounds  of  a  test,  system  performance  prediction  methods,  and  the 
expanded  use  of  synthetic  test  environments  in  lieu  of  live  tests  [GF02].  Numerous  VV&A 
issues  included: 

•  The  continuing  perception  that  VV&A  is  too  costly, 

•  VV&A  not  adequately  integrated  in  the  M&S  process, 

•  Programs  lack  independent  V&V, 

•  VV&A  lacks  up-front  estimation  of  the  effort,  causing  unnecessary  work, 

•  VV&A  lacks  multidisciplinary  membership;  and 

•  The  model-test-model  and  model  reuse  paradigms  had  not  evolved  as  anticipated 
[GF02], 

The  workshop  findings  according  to  [GF02]  also  noted  the  failure  of  De¬ 
partment  M&S  users,  including  decision-makers  to  understand  the  problem  or  question  un¬ 
der  review,  contributing  to  the  unfocused  use  of  M&S,  unbounded  VV&A,  and  the  unnec¬ 
essary  expenditures  of  resources.  [GF02]  proposed  examining  options  for  program  office 
incentives  supporting  effective  M&S  use,  improved  reuse,  and  development  of  M&S  for 
cross-program  use. 
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b.  The  Department’s  Training  Domain 

The  Department  uses  M&S  across  a  broad  spectrum  of  training  and  educa¬ 
tion  from  individual  platform  simulators  to  Joint  Task  Force  operational-  and  strategic- 
level  exercises  employing  multiple  simulations  replicating  a  theater  of  war  [DoD03a]. 
Success  of  the  Department’s  transfonnation  process  also  depends  upon  the  transfonnation 
of  the  current  Department  training  paradigm  [OPV+02]  as  directed  by  the  Defense  Plan¬ 
ning  Guidance  [DPG02]  for  fiscal  years  2003-2007.  Achieving  the  strategic  goal  of  a  train¬ 
ing  transfonnation  requires  a  common  operational  architecture  providing  interoperability  of 
live,  virtual  and  constructive  training  and  mission-rehearsal  systems  “to  build  unparalleled 
military  capabilities,  which  are  knowledge-superior,  adaptable,  and  lethal”  [OPV+02]. 

M&S  also  has  the  potential  to  greatly  improve  critical  Department  program 
initiatives  involving  joint/coalition  doctrine  development,  training,  operations,  force  man¬ 
agement,  requirements  development,  materiel  readiness,  life  cycle/  total  ownership  costs, 
human  behavior,  education,  mission  planning,  research  and  development,  modernization, 
analysis,  contracting,  and  joint  warfare  experimentation,  although  at  this  point  a  scientifi¬ 
cally  definitive  conclusion  on  its  effectiveness  cannot  be  drawn. 

c.  The  Department’s  Operations  Domain 

Department  M&S  supports  deliberate  planning,  crisis  course  of  action 
analysis,  and  mission  rehearsal.  The  Department’s  draft  Modeling  and  Simulation  Strate¬ 
gic  Plan  2003-2012  [DoD03a]  provides  a  challenging  vision  statement  for  Department 
M&S  efforts  citing  the  need  to:  “Provide  the  Power  to  predict,  Prepare,  and  Respond  Rap¬ 
idly  to  Win  Decisively  Against  Any  Threat”  [DoD03a].  Software  and  information  support 
the  goal  of  twenty  first  century  military  decision  superiority  identified  by  [Rum02b]  for 
U.S.  forces  to  communicate  with  each  other,  share  infonnation,  and  see  the  same,  precise, 
real-time  picture  of  the  battlespace,  while  simultaneously  sharing  knowledge  of  friendly 
and  enemy  locations. 

Furthermore,  the  Department  determined  that  M&S  “has  the  potential  to 
significantly  enhance  the  military  capability  and  readiness  of  US  forces”  [DoDOla],  In  or- 
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der  to  achieve  these  goals,  “DoD  must  also  undertake  high-fidelity  transformation  exercises 
and  experiments  that  address  the  growing  challenge  of  maintaining  space  control  or  de¬ 
fending  against  attacks  on  the  U.S.  national  information  infrastructure”  [RumOl].  Improv¬ 
ing  the  Nation’s  missile-defense  capability  is  a  major  pillar  of  the  National  Security  Strat¬ 
egy  [Bus02d]  and  Homeland  Defense  [Bus02c],  Capability-based  acquisition  [DoD03b, 
DoD03c]  places  additional  requirements  on  future  M&S  requirement  including  the  need  to 
model  a  potential  adversary’s  limitations  and  strengths.69 

There  is  also  an  increasing  Department  demand  for  by  the  Department1  s 
warfighting  community  for  credible  and  interoperable  new  modeling  and  simulation  pro¬ 
jects  supporting  the  development  of  the  four  major  Joint  Vision  2020  [SheOOa]  operational 
concepts  -  dominant  maneuver,  precision  engagement,  focused  logistics,  and  full  dimen¬ 
sion  protection79  [SheOOa],  linked  by  three  critical  interdependent  factors:  interoperability, 
innovation  and  decision  superiority.  Interoperability,  based  on  open  systems  architecture, 
is  the  keystone  of  future  military  operations  and  “the  foundation  of  effective  joint,  multina¬ 
tional  and  interagency  operations”  [SheOOa].  Decision  superiority  is  “. .  .superior  informa¬ 
tion  converted  to  superior  knowledge  to  achieve. .  .better  decisions  arrived  at  and  imple¬ 
mented  faster  than  an  opponent  can  react”  [SheOOa].  The  concept  of  military  decision  su¬ 
periority  has  roots  in  the  very  genesis  of  warfare.  From  warfare’s  inception  the  world’s 
military  forces  continually  refined  decision  superiority  to  achieve  a  strategic  advantage  or 
victory  in  battle.  In  the  future,  M&S  will  provide  real-time  decision  support  during  mis¬ 
sion  execution  [DoD03a] 

A  recent  Department  initiative,  the  Command,  Control,  Computer,  Commu¬ 
nication,  Intelligence,  Surveillance  and  Reconnaissance  (C4ISR)  Information  Superiority 
Modeling  and  Simulation  Master  Plan  for  improving  information  and  decision  superiority, 
plans  for  environments  “constructed,  to  the  extent  possible,  from  affordable,  reusable  com- 

69 

The  scope  of  this  effort  includes  the  adversary’s  Political,  Military,  Economic,  Social,  Infrastructure  and  Information  (PMESII) 
[DoD03a]. 

70 

Dominant  Maneuver  is  the  ability  of  joint  forces  to  gain  positional  advantage  with  decisive  speed  and  overwhelming  operational 
tempo  in  the  achievement  of  assigned  tasks.  Precision  Engagement  is  the  ability  of  joint  forces  to  locate,  surveil,  discern,  and  track 
objectives  or  targets;  select,  organize,  and  use  the  correct  systems;  generate  desired  effects;  assess  results;  and  reengage  with  decisive 
speed  and  overwhelming  operational  tempos  as  required,  throughout  the  full  range  of  military  operations.  Focused  Logistics  is  the  ability 
to  provide  the  joint  force  the  right  personnel,  equipment,  and  supplies  in  the  right  place,  at  the  right  time,  and  in  the  right  quantity,  across 
the  full  range  of  military  operations.  Full  Dimension  Protection  is  the  ability  of  the  joint  force  to  protect  its  personnel  and  other  assets 
required  to  decisively  execute  assigned  tasks  [SheOOa]. 
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ponents  interoperating  through  an  open  systems  architecture”  [DoD02d].  [SH96]  proposed 
organizing  around  information  enabling  distributed  operations  with  a  decentralized  deci¬ 
sion-making  process,  as  a  means  to  establish  the  environment  for  quality  and  speed.  How¬ 
ever,  issues  with  M&S  reuse  and  interoperability  in  the  Department  indicate  that  the  “value 
of  past  investments  in  M&S  is  depreciated  because  the  DoD  does  not  have  the  capability  to 
easily  identify  algorithms,  data  sets,  models  or  M&S  that  can  be  reused”  [DoDOla]. 

The  Department  requires  additional  resolution,  level  of  detail  and  refine¬ 
ment  to  existing  M&S  tools,  especially  for  opposing-force  representations  of  entities, 
which  “must  be  significantly  increased  to  pennit  realistic  assessments  of  C4ISR  perform¬ 
ance  [and]  the  degree  of  fidelity  and  detail  of  M&S  of  C4ISR  functions  be  commensurate 
with  this  importance”  [DoD02d].  The  Department  also  requires  increased  fidelity  and 
M&S  with  “more  accurate  representations  of  all  elements  of  the  mission  space”  [DoDOla]. 
Although  the  Department  made  progress  in  the  area  of  developing  authoritative  representa¬ 
tions,  [DoDOla]  notes  “many  combat,  combat  support,  and  combat  service  support  systems 
abound,  making  it  difficult  to  determine  which  are  authoritative  or  best  of  breed.” 
[DoDOla], 


d.  The  Department’s  Analytical  Domain 

The  military  operations  research  community  advanced  operational  decision¬ 
making  over  the  past  fifty  years  [You97  MK98],  employing  computer-based  models  and 
software  M&S  as  “tools  supporting  an  analytical  process  with  better  decisions  being  the 
objective”  [Hug97b].  [Pre97]  suggests  that  software  supports  a  multitude  of  uses,  includ¬ 
ing  M&S,  and  delivers  infonnation,  which  may  be  the  most  critical  product  of  the  twenty- 
first  century.  Analysis  supported  by  M&S  occurs  in  every  Department  domain.  In  addi¬ 
tion,  the  Department  uses  M&S  in  analysis  or  wargames  supporting  force  structure  analysis 
and  long-range  strategic  planning,  both  major  Department  activities. 
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e.  The  Department's  Experimental  Domain 

The  Department  also  uses  M&S  to  explore  new  capabilities,  portray  future 
operational  environments  and  develop  new  tactics,  techniques,  procedures,  doctrine,  organ¬ 
izational  structures,  processes,  and  systems  [DoD03a].  The  Revolution  in  Military  Affairs 
initiatives  employ  large-scale,  legacy  M&S,  future  evolutions  of  legacy  M&S  software  sys¬ 
tems,  and  new  generations  of  live,  virtual  and  constructive  DoD  M&S,  for  experimentation, 
advanced-concept  exploration,  analysis,  and  requirements  elicitation.  These  initiatives 
support  expanded  joint,  combined,  or  multinational  federations,  enable  M&S-based  acqui¬ 
sition,  and  advance  development  of  distributed  system  engineering  collaborative  environ¬ 
ments. 

Department  M&S  users  require  timely  simulation  applications,  improved 
analysis  capabilities,  and  improved  operational  support  with  M&S  integrated  with  go-to- 
war  systems,  including  international  collaborative  initiatives  and  coalition  operations 
[DoD03a].  [RumOl]  cited  M&S  in  the  2001  DoD  Quadrennial  Defense  Review  as  critical 
for  future  joint,  combined  and  coalition  operations  and  the  dynamic  military  transfonnation 
process  to  the  point  where  “U.S  forces  rely  heavily  on  war  games  and  M&S  to  support  this 
program  of  field  exercises  and  experiments”  [RumOl]. 

Millennium  Challenge  2002  (MC02),  the  most  elaborate  war  game  ever 
conducted  by  the  Department,  involved  13,500  joint  service  participants,  federated  forty- 
two  models  [Koz02]  in  seventeen  locations  and  nine  live  training  sites,  and  concluded  three 
weeks  of  playing  a  classified  2007  scenario  with  specified  objectives  [MC02],  on  August 
15,  2002,  after  taking  two  years  and  $250  million  to  plan  and  execute  [Nur02,  Nay02]. 
Lessons-leamed  from  MC02  impacting  M&S  credibility  and  confidence  of  results  identi¬ 
fied  in  [Nay02,  Sch02b]  includes  recommendations  for  a  better  Department  doctrine  defin¬ 
ing  how  to  achieve  value  from  the  growing  portfolio  of  the  Department’s  M&S,  independ¬ 
ent  reviews  of  the  experiments,  a  better  definition  between  objectives  designed  to  validate 
concepts  and  objectives  designed  to  identify  victors  of  the  virtual  battles,  and  an  increased 
emphasis  on  effectiveness,  including  smaller  games  and  M&S  to  test  and  train. 
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D.  SUMMARY 

Chapter  II  provided  a  literature  survey  on  the  current  research  areas  relevant  to  De¬ 
partment  simulation  credibility.  The  literature  survey  clearly  revealed  two  major  eras  af¬ 
fecting  Department  simulation  credibility.  The  Growth  Era  before  1994  experienced  over 
forty  years  of  research,  experimentation,  and  development  of  a  wide  variety  and  types  of 
M&S  with  varying  degrees  of  credibility.  The  Managed  Era,  or  post- 1994  period,  intro¬ 
duced  policy  and  management  procedures  as  the  Department  began  to  plan,  program,  de¬ 
velop  policy  and  procedures,  budget,  and  corporately  manage  the  Department  Enterprise¬ 
wide  M&S  portfolio  developed  during  the  Growth  Era  and  plan  for  new  joint-simulations 
to  replace  many  of  the  legacy  Growth  Era  M&S.  This  chapter  also  introduced  the  Depart¬ 
ment’s  M&S  management  framework  for  establishing  credible  simulations,  including 
classes  of  simulation,  simulation  hierarchies,  and  categories  of  M&S  (live,  virtual,  con¬ 
structive),  within  a  prescribed  hierarchy  of  M&S  (engineering,  engagement,  mission,  and 
campaign). 

The  current  Department  M&S  hierarchy  maintains  only  general,  qualified  levels  for 
M&S  fidelity  and  resolution,  provides  only  limited  support  for  the  user  to  ascertain  a  simu¬ 
lation’s  attributes.  The  model  provides  little  if  any  support  for  quantifiable  measures  of 
fidelity  and  resolution.  In  addition,  metrics  based  on  this  model  tend  to  be  qualified,  sub¬ 
jective,  and  lack  authority.  In  this  research  we  analyzed  the  current  Department  M&S  Hi¬ 
erarchical  model  as  a  core  asset  and  reviewed  alternatives  for  mining  reverse  engineering 
and  reengineering  these  core  assets  into  product  line  architecture. 

Other  Department  initiatives  included  the  establishment  of  new  distributed  simula¬ 
tion  capabilities  supporting  the  expanding  need  of  credibility  by  the  major  stakeholders. 
Major  categories  of  stakeholders  influencing  the  Department’s  M&S  management  priorities 
included  the  acquisition,  training,  operations,  analysis,  and  experimental  domains. 

Research  indicated  that  M&S  credibility  was  a  multi-dimensioned  problem.  Chap¬ 
ter  III  through  Chapter  VII  review  different  dimensions  of  the  Department’s  M&S  credibil¬ 
ity  challenge.  A  taxonomy  provided  at  Figure  2-1  represents  the  synthesis  of  the  major  ar¬ 
eas  identified  in  the  literature  search  to  counter  these  major  challenges  and  support  credibil- 
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ity  in  M&S  and  confidence  in  the  results  they  produce,  including  the  architectural  frame¬ 
work,  VV&A,  software  engineering,  process  improvement,  and  knowledge  acquisition. 
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III.  CREDIBILITY  OF  DEPARTMENT  SYSTEM  REPRESENTATIONS 


A.  INTRODUCTION 

Chapter  III  introduces  M&S  credibility  issues  and  identifies  concerns  that  decision¬ 
makers  have  with  Department  simulation  model  representation  and  results,  followed  by  an 
overview  of  the  Departments  VV&A  processes,  and  several  factors  contributing  to  the  im¬ 
plementation  of,  or  shortfalls  of  the  VV&A  processes  in  the  Department.  Two  major  peri¬ 
ods,  the  Growth  and  the  Managed  Eras  characterize  the  Department’s  experiences  with 
M&S.  The  Growth  period  included  the  forty  years  of  M&S  development  prior  to  1994,  and 
the  Managed  Era  includes  the  Department’s  post- 1994  efforts  to  improve  M&S  credibility. 

B.  THE  GROWTH  ERA:  M&S  CREDIBILITY  ISSUES  IDENTIFIED 

In  1978,  a  United  States  Comptroller  General  report  [PAD78]  identified  a  number 
of  major  systemic  quality  issues  in  the  simulation  software  development  practices  of  gov¬ 
ernment  agencies  and  concluded  that: 

There  are  no  generally  accepted  guidelines  for  establishing  models  credibil¬ 
ity.  In  addition,  there  is  not  a  generally  accepted  threshold  beyond  which  a 
model  can  be  termed  ‘credible’,  and  the  concept  of  credibility  varies  greatly 
between  model  builders/developers  and  users/decision-makers  as  well  as 
among  individuals  within  either  of  these  two  broad  groups  [PAD78], 

The  need  for  credible  M&S  grew  in  the  Nation’s  private  and  public  sectors.  By 
1980,  infonnation  from  computer-based  simulations  played  an  expanded  role  in  the  analy¬ 
sis  of  public  policy  issues  and  supported  decisions  for  systems  costing  billions  of  dollars 
[Sta74,  PAD78,  PAD79a,  PAD79b,  PAD80,  Che86a,  Che86b,  Che87].  Computer-based 
simulations  also  aided  political-military  analysis  [DBW87,  DH92a,  DH92b],  supported  the 
development  of  strategic  course  of  action  development  underpinning  national  policy 
[Lie97],  assisted  elements  of  the  Department’s  decision-making  process  [HRA+01],  and 
recently  emerged  in  an  expanded  role  in  the  United  States  National  Security  Strategy 
[Bus02d]. 

The  report  Models,  Data,  and  War:  A  Critique  of  the  Foundation  for  Defense 
Analysis,  [PAD80]  found  the  same  major  systemic  shortcomings  in  Department-wide  M&S 
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cited  earlier  by  the  General  Accounting  Office  (GAO)  in  other  government  agencies  in¬ 
cluding  the  GAO  [PAD78],  the  Federal  Energy  Commission  [PAD79a],  and  a  survey  of 
other  government  agencies  [PAD79b]: 

•  Overall  poor  quality, 

•  Lack  of  documentation, 

•  Inadequate  understanding  of  model  assumptions,  limitations,  and  capabilities, 

•  Insufficient  model  development  practices, 

•  The  need  for  better  coordination  between  the  model  developer  and  the  user, 

•  The  requirement  for  better  monitoring  of  development  efforts, 

•  The  need  for  improved  procedures  to  update  and  maintain  models, 

•  Improve  how  the  M&S  credibility  is  established, 

•  Improve  how  the  validity  of  the  model  is  detennined, 

•  Problems  with  finding  data  to  make  the  model  function, 

•  Models  supporting  credible  Department  decisions  should  be  transparent,  ap¬ 
praised71,  and  consistent  [PAD78,  PAD79a,  PAD79b,  PAD80]. 

However,  by  1988  the  Department  realized  that  the  existing  ways  and  means  for 
determining  simulation  credibility  were  insufficient  and  improving  simulation  credibility 
and  user  confidence  in  simulation  results  would  require  new  policies,  procedures,  re¬ 
sources,  and  organizations.  A  Secretary  of  Defense  memorandum  [Car88]  noted  that,  “as 
the  need  to  improve  the  capabilities  and  credibility  of  simulation  continues  to  increase,  we 
are  redoubling  our  efforts  to  develop  comprehensive,  consistent  and  credible  guidance  to 
the  services”  [Car88],  The  Director,  Operational  Test  and  Evaluation,  stated  that  when  “ 
results  from  modeling  and  simulation  are  used... special  care  is  necessary  to  ensure  they  are 
credible”  [Kri89].  Early  evaluations  of  simulations  used  in  operational  tests  defined  simu¬ 
lation  credibility  as  a,  “fragile  commodity... [that]  as  applied  to  the  M/S  processes  and  re¬ 
sults,  is  a  combined  impression  of  the  inputs,  processes,  outputs,  conclusions,  the  persons 
or  agencies  involved,  and  the  strength  of  the  evidence  presented”  [DOT89]. 

C.  THE  MANAGED  ERA  FOR  CREDIBLE  DEPARTMENT  M&S 

In  response  to  the  identified  systemic  deficiencies,  the  Department  implemented 
and  updated  several  M&S  management  initiatives  including  M&S  management  policy 


The  term  ‘appraised’  in  this  context  means  the  model  is  mathematically  correct,  matches  the  real  world,  and  uses  empirically  valid 
data  [PAD80]. 
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[DoD94],  master  plans  [DoD95,  DoDOla],  and  verification72,  validation73,  and  accredita¬ 
tion74  instructions  and  procedures  [DoD96,  GMS+96,  DoDOOa,  RPGOO,  DoDOlb, 
DoD02a],  including  VV&A  of  conceptual  models,  and  data75,  in  order  to  resolve  these  is¬ 
sues.  The  Department  published  the  first  Department  M&S  management  directive,  DoD 
Modeling  and  Simulation  Management  [DoD94]  in  January  1994  to  resolve  systemic  is¬ 
sues,  provide  departmental  M&S  policy,  detennine  organizational  responsibilities,  define 
information  requirements76,  establish  an  investment  program,  and  develop  procedures,  in¬ 
cluding  VV&A,  to  “Strengthen  the  uses  of  M&S  in  the  Department  of  Defense”  [DoD94], 
[DoD94]  also  assigned  Department  M&S  Executive  Agents  (MSEAs)  responsibilities  to 
DoD  Components  for  common77  and  general-use78  M&S  applications  such  as  terrain  or  en¬ 
vironment. 

However,  major  systemic  shortcomings  persisted  to  undermine  confidence  in  simu¬ 
lation  results  when  the  Department  published  the  first  Department  M&S  master  plan,  DoD 
Modeling  and  Simulation  Master  Plan  [DoD95],  At  the  time,  M&S  were  still  too  costly, 
failed  to  meet  users  needs,  remained  stove-piped  within  the  functional  communities,  ex¬ 
perienced  excessive  development  time,  lacked  credible  VV&A  procedures,  and  needed  im- 
provements  in  many  quality  characteristics:  interoperability  ,  security,  maintainability,  ex¬ 
tensibility,  and  usability  [DoD95]. 

Concurrently,  a  1995  Modeling  and  Simulation  Benefits  Task  Force  [WSM+96] 
completed  a  limited  review  of  the  Department’s  corporate  M&S  assets  to  document  the 
quantifiable  benefits  of  the  Department’s  investment  in  M&S.  Based  upon  a  very  limited 
sample  size  and  meta-analysis,  [WSM+96]  achieved  mixed  results  and  concluded  that:  “a 

72 

Verification  is  the  process  of  determining  that  a  model  or  M&S  implementation  accurately  represents  the  developer’s  conceptual 
description  and  specification.  Verification  also  evaluates  the  extent  to  which  the  model  or  M&S  has  been  developed  using  sound  and 
established  software  engineering  techniques  [DoD95]. 

73 

Validation  is  the  process  of  determining  the  extent  to  which  a  model  or  M&S  is  an  accurate  representation  of  the  real  world  from  the 

perspective  of  the  intended  use(s)  of  the  model  or  M&S  [DoD95]. 

74  .....  . 

Accreditation  is  the  official  certification  that  a  model  or  M&S  is  acceptable  for  use  for  a  specific  purpose  [DoD95]. 

75 

Data  is  a  representation  of  facts,  concepts,  or  instruction  in  a  formalized  manner  suitable  for  communications,  interpretation,  or  proc¬ 
essing  by  humans  or  by  automatic  means  [DoD98]. 

76  “When  possible,  existing  information  systems  shall  be  used  or  modified  rather  than  creating  new  systems”  [DoD94]. 

77 

Common-use  M&S  are  applications,  including  services  or  materials,  provided  by  a  DoD  Component  to  two  or  more  DoD  components 
[DoD94]. 

General-use  M&S  applications  are  representations  used  by  or  common  to  many  M&S,  including  the  physical  environment  or  envi¬ 
ronmental  effects  such  as  terrain,  atmosphere,  or  hydrographic  effects  [DoD94]. 

79 

Interoperability  is  the  ability  of  a  model  or  simulation  to  provide  services  to,  accept  services  from,  other  models  and  simulations,  and 
to  use  the  exchanged  services  to  operate  effectively  together  [DoD95]. 
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formal  reporting  mechanism  does  not  exist  for  gathering  information,  nor  do  methodolo¬ 
gies  exist  for  objectively  assessing  the  value  of  using  M&S”  [WSM+96], 

A  1996  DoD  Study  on  the  Effectiveness  of  Modeling  and  Simulation  in  the  Weapon 
System  Process  [PHP+96]  identified  three  major  types  of  M&S  challenges:  technical,  cul¬ 
tural  and  managerial,  and  fifteen  specific  issues,  many  previously  cited  by  [PAD78, 

PAD80,  DoD95].  Many  of  the  systemic  challenges  identified  by  [PHP+96]  addressed 
credibility  concerns  with  issues  including  proprietary  models  and  data,  security  of  data, 
availability  of  data  descriptions,  interoperability  of  M&S  tools,  consistent  variable  resolu- 
tion  descriptions,  physics-based  models  based  on  empirical  data  rather  than  physical  prin¬ 
ciples,  and  continued  VV&A  deficiencies. 

[PHP+96]  also  cited  proprietary  data,  and  resistance  to  funding  V&V  as  two  M&S 
management  problems  in  the  Department.  The  Department  also  acknowledged  the  limited 
research  and  review  of  V&V  methodologies  within  the  community  and  that  the  “verifica¬ 
tion  and  validation  (V&V)  process  leading  to  accreditation  is  expensive  and  not  well  un¬ 
derstood”  [PHP+96].  In  addition,  the  complexity  of  the  software  engineering  process  of 
non-trivial  software-intensive  simulations  and  establishment  of  M&S  credibility  [RosOl, 
WPL02]  add  difficulty  to  the  validation  process  and  may  limit  credible  validation  efforts 
[KMOla]. 

These  challenges  highlighted  the  lack  of  “broadly  accepted  community  standards 
for  representing  military  systems  and  organizations  in  M&S”  [DoD95].  Consequently,  rep¬ 
resentations  of  the  same  system  in  different  models  were  frequently  incompatible  according 
to  [DoD95],  limiting  the  effectiveness  of  the  federation  objectives  and  reducing  confidence 
in  the  results  as  a  consequence  of  unresolved  substantive  interoperability  issues.  To  im¬ 
prove  M&S  credibility,  [DoD95]  established  six  Department-wide  M&S  objectives  with 
seventeen  sub-objectives,  and  one  hundred  and  fifteen  supporting  action.  The  six  major 
Department  M&S  objectives  delineated  by  [DoD95]  established  requirements  to: 


80  Resolution  is  the  degree  of  detail  and  precision  used  in  the  representation  of  real-world  aspects  in  a  model  or  simulation  [DoD95]. 
Resolution:  1 .  The  degree  of  detail  used  to  represent  aspects  of  the  real  world  or  a  specified  standard  or  referent  by  a  model  or  simula¬ 
tion.  2.  Separation  or  reduction  of  something  into  its  constituent  parts;  granularity  [RPG00]. 
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•  Develop  a  common  technical  framework  (CTF)  for  M&S  with  sub-objectives  for 
HLA  conceptual  models  of  the  mission  space81  (CMMS),  and  data  standardization, 

•  Develop  timely  /  authoritative  representations  of  the  natural  terrain, 

•  Develop  authoritative  representations  of  systems, 

•  Develop  authoritative  representations  of  human  behavior, 

•  Establish  a  M&S  infrastructure  to  meet  developer  /  end-user  needs  with  a  VV&A 
sub-objective, 

•  Share  the  benefits  of  M&S  [DoD95]. 

Acknowledging  the  criticality  of  computer-based  infonnation  [JBC+99]  stated: 

“Software  is  the  physical  structure  of  the  infonnation  age...  It  is  fundamental  to  economic 

success  and  national  security.”  However,  here  may  be  even  more  challenges,  as  the  bipolar 

threats  of  the  late  twentieth  century  fade  way  and  are  replaced  by  the  modern  twenty-first 

century  world  with  a  multitude  of  military  asymmetric  threats,  [Lie97]  cautions  that  it  is: 

Now  necessary  to  address  a  much  more  confusing  set  of  problems:  unex¬ 
pected  environments,  non-traditional  forces,  new  missions,  and  new  con¬ 
cepts  of  operation.  Relevant  data  are  limited.  Many  of  the  widely  used 
models  of  the  last  40  years  or  so  may  be  inappropriate  for  the  new  arena 
|  I.ic97|. 

Trying  to  overcome  over  forty  years  of  modeling  and  simulation  history  without 
V&V  guidance,  while  maintaining  a  large  inventory  of  legacy  models,  and  attempting  to 
stay  ahead  of  the  learning  curve,  the  Department  developed  the  first  VV&A  instruction, 
DoD  Modeling  and  Simulation  (M&S)  Verification,  Validation,  and  Accreditation  (VV&A) 
[DoD96],  and  VV&A  best  practices  guide,  Department  of  Defense  Verification,  Validation, 
and  Accreditation  (VV&A)  Recommended  Practices  Guide  [GMS+96],  In  addition  the  De¬ 
partment  continuously  updated  M&S  policy  [Gan98,  Gan99,  WolOl],  verification  and  vali¬ 
dation  instructions  [DoDOOa,  DoDOlb,  DoD02a],  masterplans  [DoDOla,  DoD02c, 
DoD02d],  manuals  [DoD98],  verification  and  validation  technical  guidance  [GMS+96, 
RPG00]  and  developed  a  strategic  plan  [DoD03a]  to  support  these  objectives.  Department 


Designated  as  the  Conceptual  Model  of  the  Mission  Space  (CMMS)  in  the  [DoD95],  The  term  CMMS  is  in  the  process  of  officially 
being  changed  to  the  Functional  Description  of  the  Mission  Space  (FDMS)  [RPG00]. 


45 


82 

components  ~  also  established  supplemental  policies  and  procedures  [AP93,  AR97, 
DON99,  AFIOO], 

The  Department’s  first  strategic  plan  [DoD03a],  provides  six  strategic  goals  with 
twenty-nine  objectives  supporting  the  new  M&S  mission  and  vision  statements  to  help  the 
Department  conduct  war,  transform  the  force,  strengthen,  Joint  warfighting  capabilities, 
develop  and  test  new  concepts  to: 

•  Operationalize  M&S  as  a  real-time  decision  support  tool  for  the  warfighter  (five 
supporting  objectives), 

•  Provide  rapidly  and  readily  accessible  and  executable  M&S  (six  supporting  objec¬ 
tives), 

•  Increase  M&S  awareness,  education,  training,  and  collaboration  (four  objectives), 

•  Effectively  represent  the  complexity  of  modem  warfare  across  the  full  spectrum  of 
operations  (five  supporting  objectives), 

•  Employ  M&S  to  accelerate  acquisition,  reduce  life-cycle-costs,  foster  interoperabil¬ 
ity,  and  improve  quality  of  systems  (five  supporting  objectives), 

•  Align  and  increase  science  and  technology  investment  to  transform  M&S  for  the 
warfighter  (four  supporting  objectives)  [DoD03a]. 

The  Department  and  the  Service  components  also  developed  M&S  guidelines  and 
best  practices  [PMR94,  GMS+96,  Fal97,  SAFOO]  to  improve  confidence  by  decision¬ 
makers  that  Department  M&S  provides  reasonable,  correct,  defendable  and  credible  results 
for  given  situations,  circumstances  and  environments.  The  private  and  technical  sector, 
including  the  Simulation  Interoperability  Standards  Organization  (SISO),  the  International 
Standards  Organization  (ISO),  and  the  IEEE  also  contributed  to  the  growing  M&S  body  of 
knowledge.  In  addition  to  the  development  of  policies,  plans,  procedures,  instructions,  and 
guidance  the  Department  established  the  Defense  Modeling  and  Simulation  Office 
(DMSO)  to  plan,  program,  coordinate  and  budget  the  Department’s  M&S  program  and  an 
Executive  Council  for  Modeling  and  Simulation  (EXCIMS)  to  provide  assistance  and  ad¬ 
vice  on  major  M&S  issues  [DoD94]. 

The  Department  also  made  significant  investments  to  improve  the  credibility  in 
M&S  during  the  1990s.  These  major  investment  initiatives  includes  the  common  technical 
framework  initiatives  for  HLA,  development  of  a  HLA-compliant  Runtime  Infrastructure 


82  DoD  Components  include  the  Office  of  the  Secretary  of  Defense  (  OSD)  Components,  the  Military  Departments,  the  Chairman  of  the 
Joint  Chiefs  of  Staff,  the  Combatant  Commands,  the  General  Counsel  of  the  Department  of  Defense,  the  Inspector  General  of  the  De¬ 
partment  of  Defense,  the  Defense  Agencies,  and  the  DoD  Field  Activities  [DoD94], 
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(RTI),  authoritative  data  sources,  including  the  Environmental  Effects  for  Distributed  In- 
teractive  Simulation  (E"DIS)  program  [PBG+94],  and  the  Synthetic  Environment  Data 
Representation  and  Interchange  Specification  (SEDRIS)  project,  and  major  joint  simulation 
development  efforts  such  as  the  JSIMS,  JWARS,  and  JMASS  initiatives. 

D.  ESTABLISHING  DOD  M&S  CREDIBILITY  AND  USERS  CONFIDENCE 

The  Department  established  the  M&S  VV&A  process  to  develop  credibility  in 
M&S  software  and  maintain  user  confidence  in  the  results.  The  Department  and  the  private 
sector  M&S  community  developed  a  body  of  research,  studies,  best  practices,  and  analysis 
focused  on  improved  verification  and  validation  theories,  methods,  techniques,  tools,  and 
procedures  to  remedy  systemic  issues  and  shortcomings  identified  by  reports,  studies,  and 
other  assessments  cited  earlier. 

Research  into  the  Department’s  VV&A  process  identified  several  patterns  between 
the  individual  VV&A  processes  and  several  other  areas  affecting  the  establishment  of 
M&S  credibility.  The  areas  identified  in  Figure  3-1  address  topics  consistently  raised  as 
concerns  and  issues  in  the  VV&A  literature  and  provides  a  synthesized  M&S  credibility 
taxonomy  addressed  in  the  following  chapters: 

•  The  Department’s  verification  process, 

•  The  Department’s  validation  process, 

•  The  Department’s  accreditation  process, 

•  The  role  of  data  in  the  VV&A  process, 

•  Verification  and  validation  techniques, 

•  Shortfalls  of  the  Department’s  VV&A  process, 

•  Software  development  /  software  engineering  practices, 

•  The  Department’s  M&S  framework, 

•  Heterogeneous  data, 

•  VV&A  practices, 

•  Organizational  areas  and  issues, 

•  The  Department’s  Common  Technical  Framework, 

•  VV&A  techniques, 

•  Software  architecture, 

•  Quality, 

•  Process  Maturity, 

•  Secondary  contributing  factors  (e.g.  reuse,  documentation,  risk  management,  cost, 
return  on  investment), 
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Department  institutional  constraints. 


Figure  3-1.  Synthesized  M&S  Credibility  Taxonomy 

E.  VERIFICATION,  VALIDATION  &  ACCREDITATION  PROCESSES 

Early  studies  by  [Arm64,  Mec64,  and  NF67]  identified  the  lack  of  verification 
methods  in  the  M&S  process  and  proposed  the  concepts  of  high  face  validity,  conceptual 
validity,  and  operational  validity.  Later,  [Mih72,  Sha75]  addressed  the  concepts  of  V&V  in 
conjunction  with  systems  analysis,  system  synthesis  and  model  analysis.  [Sar91,  Sar92, 
Sar98]  considers  M&S  V&V  critical  and  addressed  model  validation  methods  and  tech¬ 
niques  including  conceptual  model  validity,  model  verification,  operational  validity,  data 
validity  and  development  of  a  “model  range  of  accuracy”  from  techniques  employed  in 
M&S  output  analysis. 

[MBZ99]  provided  a  concise  chronology  of  M&S  credibility  from  1962  through  the 
1990’s  and  identified  the  growing  emphasis  and  research  on  V&V,  and  the  growing  aware¬ 
ness  of  the  importance  of  the  underlying  software  development  foundation.  In  1988,  the 
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Director,  OTE  acknowledged  the  emerging  efforts  to  establish  Department  M&S  verifica¬ 
tion,  validation  and  credibility  assessments,  noting  that  it  was  “essential  that  the  M&S  em¬ 
ployed  and  the  results  derived  from  them  be  credible”  [Kri88]. 

There  is  a  significant  body  of  related  literature  on  improving  M&S  credibility  and 
improving  confidence  in  simulation  results.  A  large  body  of  simulation-related  topics  in¬ 
cluding  risk  reduction,  V&V  principles  and  techniques,  M&S  quality  characteristics,  proc¬ 
ess  improvement,  software  development  /  engineering  procedures,  domain  implementation 
uses,  and  approved  architectural  standards  for  achieving  higher  M&S  confidence  levels, 
includes  research  by  [BL91,  Sad91,  Dow92,  Gia92,  Hen92,  Met92,  Bec94,  Lew94,  Par94, 
Cau95,  Gan95,  BCN96,  CCK+96,  Por96,  FY97,  ML97,  Mue97a,  Mue97b,  Pre97,  Bar98, 
Che98,  Gan98,  Gau98,  HM98,  JDD98,  LR98,  MT98,  NM98,  NMY+98,  Pac98c,  Roa98, 
ROG98,  BSW99,  Dah99,  DMS99,  FGW+99,  GB99,  Gre99,  Mar99,  Pac99a,  San99, 

Whi99,  BFOO,  BSL+00,  COHOO,  EKHOO,  HNC+00,  MLU+00,  SanOO,  BS01,  CooOla, 
CY01,  DDH01,  FED01,  FH01,  HalOlb,  GKB01,  HNK+01,  LS01,  LuqOl,  MJL01,  MorOl, 
PacOlb,  PacOlc,  PugOl,  RosOl,  YPH01,  BML02,  Bre02a,  CW02,  Dan02,  FRC+02,  Kil02, 
Pac02b,  Pac02c  You02a,  You02b]. 

More  recently,  [CoyOOb  and  BG01]  noted  several  significant  systemic  M&S  chal¬ 
lenges  within  the  DOD  Test  and  Evaluation  community  affecting  confidence  in  the  results. 
[CoyOOb]  specifically  cautioned  that  the  use  of  M&S  to  extrapolate  beyond  the  known 
bounds  almost  never  works,  however,  interpolating  between  two  demonstrated  test  results 
is  an  acceptable  practice  in  evolutionary  acquisition  programs.  [Tho97a]  also  raised  the 
concern  that  the  V&V  process  lacked  adequate  definition  as  recently  as  1983. 

Current  V&V  theories,  methods,  techniques,  and  procedures  cover  a  wide  range  of 
activities  cited  by  [Kri89,  RSH90,  Rit92,  WS92,  WF93,  ARS+96,  JAS97a,  JAS97b, 
Pac98d,  RPGOO,  Roa98,  SMOO,  RakOl,  WriOla,  WH02]  and  techniques  proposed  by 
[Gia92,  Cau95,  FH01,  FriOl,  GMS+96,  KFC98,  MM98,  RPGOO].  Verification  and  valida¬ 
tion  for  large  complex  M&S  such  as  mission-level  models  or  physics  based  models  have 
additional  impediments  [KMOla]  including  the  dynamic  nature  of  M&S  evolution,  the  cost 
and  constraints  of  the  data,  the  difficulty  of  validating  the  models,  and  the  limited  shelf  life 
of  previous  validation  efforts. 
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The  Department  published  [GMS+96],  and  established  the  twelve  guiding  princi¬ 
ples  for  the  VV&A  of  M&S  and  federations.  [Bal98]  also  contributed  fifteen  guiding  prin¬ 
ciples  to  provide  better  insights  into  VV&A  activities,  supporting  the  Department’s  model 
[GMS+96]  or  the  [Bal98]  M&S  life  cycle  models.  A  follow-on  Internet  variant  of  the 
[GMS+96],  the  W&A  Recommended  Practice  Guide  [RPGOO],  promotes  the  current  best 
V&A  practices  and  procedures  for  legacy  M&S  (Figure  3-2). 


PROBLEM  SOLVING  PROCESS 
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Figure  3-2.  VV&A  and  Legacy  M&S  (From  [RPGOO]) 

Other  approaches  evolved.  [RakOl]  identified  three  basic  processes  for  software 
validation  based  on  testing,  measurement,  and  software  reliability  growth.  Current  VV&A 
procedures  built  on  these  theories  and  lessons-leamed  from  previous  VV&A  approaches, 
including  the  Aggregate  Level  Simulation  Protocol  (ALSP)  VV&A  process  [TP96]  and  the 
Distributed  Interactive  Simulation  (DIS)  VV&A  process  [GW96,  MBZ99].  [Bal98]  ad¬ 
vanced  an  analytic  hierarchy  process,  which  supported  the  measurement  and  evaluation  of 
both  quantitative  and  qualitative  factors,  expert  knowledge,  independent  evaluation  and 
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comprehensive  assessment  to  improve  user  confidence  in  M&S  results.  In  addition, 

[Bal98]  introduced  a  model-testing  concept  called  model  verification,  validation,  and  test 
(VV&T)  for  identifying  and  ascertaining  whether  specific  inaccuracies  or  errors  exist  in  the 
model.  [Bal98]  also  recommended  continuous  V&V  planning,  execution,  and  documenta¬ 
tion  throughout  the  entire  M&S  life  cycle. 

A  Modeling  and  Simulation  Infonnation  Analysis  Center  report  on  VV&A  auto¬ 
mated  support  tools  [DMSOOb]  and  a  subsequent  article  by  [GFM01]  proposed  leveraging 
existing  software  development  industry  tools,  basically  expanding  the  development  and 
application  of  V&V  automated  test  support  tools  analogous  to  those  currently  employed  in 
the  test  and  evaluation  community.  [Lew98]  proposed  a  V&V  Managers  Toolkit  based  on 
a  VV&A  management  model  with  a  metric  suite  for  cost,  performance,  schedule,  and  tech¬ 
nical  performance  indicators.  Currently,  automated  tool  support  for  the  VV&A  process  is 
extremely  limited  [JorOl].  In  addition,  [HNP97]  cautions  that  automated  tools  to  assist  the 
complex  and  difficult  validation  process  may  be  undesirable  from  an  engineering  view¬ 
point. 

Whether  using  qualitative  or  quantitative  methods,  a  combination  of  the  two  meth¬ 
ods  to  complement  each  other,  with  or  without  automated  tools,  the  V&V  process  requires 
disciplined  processes  and  procedures  to  provide  satisfactory  levels  of  confidence  in  the 
simulation  results.  However,  at  the  present  time  [RakOl]  notes,  “basic  software  V&V  ac¬ 
tivities  are  not  well  understood  and  are  applied  inconsistently”  [RakOl]. 

1.  The  Department’s  Verification  Process 

The  software  V&V  process  entails  a  broad  scope  of  activities,  which  support  soft¬ 
ware  quality  characteristics  [DB91,  Dav92c,  AV98,  HalOla]  discussed  in  Chapter  VI.  The 
verification  process  detennines  the  M&S  capability  and  establishes  the  foundation  for  con¬ 
fidence  in  the  model,  establishing  quality  early  in  the  development  process  by  ensuring  the 
developer  accurately  captured  the  validated  conceptual  model  in  the  implementation  de¬ 
sign.  “Verification,”  as  defined  by  the  Department  is  “the  process  of  determining  that  a 
model  implementation  accurately  represents  the  developer’s  conceptual  description  and 
specifications”  [DoD94,  DoD96,  DoDOla]  when  compared  with  the  first  abstraction  of  the 
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real  world,  the  conceptual  models  of  the  mission  space  (CMMS).  In  addition,  “verification 
ensures  that  a  M&S  meets  all  the  requirements  specified  by  the  user  and  that  it  implements 
those  requirements  correctly  in  software”  [GMS+96], 

There  are  many  other  definitions  for  verification.  Verification  is  also  concerned 
with  solving  the  equations  right  [Roa98].  The  IEEE  similarly  defines  verification  as  “con¬ 
firmation  by  examination  and  provisions  of  objective  evidence  that  specified  requirements 
have  been  fulfilled”  [IEE98b].  Verification  also  supports  the  implementation  of  the  func¬ 
tional  interface,  and  output  requirements  and  may  be  defined  as  “Did  I  build  the  thing  cor¬ 
rectly”  [GMS+96]? 

The  conceptual  model  in  the  M&S  process  is  analogous  to  the  keystone  in  a  build¬ 
ing  archway.  A  missing,  incorrect,  incomplete  or  sub-standard  keystone  threatens  the 
structural  integrity  of  the  archway.  Today,  [Pac99b]  notes  conceptual  model  documenta¬ 
tion  “for  many  legacy  M&S  is  limited  or  non-existent”[Pac99b].  [Ham96,  HNP97]  further 
note  that  confidence  in  the  verification  process  depends  upon  the  underlying  model’s  valid¬ 
ity.  The  absence  of  a  valid  conceptual  model  threatens  the  integrity  and  credibility  of  the 
simulation  and  confidence  in  the  simulation  results. 

2.  The  Department’s  Validation  Process 

Model  and  simulation  validation  is  performed  at  two  distinct  levels:  1)  conceptual 
model  validation,  and  2)  result  validation,  and  satisfies  the  criteria  of  how  well  the  func¬ 
tions  match  the  real  world  and  addresses  “Did  I  build  the  right  thing”[GMS+96]?  The  con¬ 
ceptual  model  validation  is  an  essential,  integral  component  of  the  Department’s  VV&A 
process  and  is  the  foundation  for  establishing  credibility,  defining  fidelity  requirements  and 
developing  accurate  model  representations.  Results  validation  compares  the  M&S  results 
with  known  or  projected  behavior,  tests,  and  sensitivity  analysis  or  subject  matter  expert 
opinion  to  detennine  if  the  results  are  “sufficiently  accurate  for  the  range  of  intended  uses 
of  the  M&S”  [GMS+96]. 

The  validation  process  complements  and  supplements  the  verification  process,  and 
supports  development  of  M&S  credibility  [IEE86,  Sar91,  Cyn92,  DH92a,  Hen92,  Sar92, 
Ham96,  Bal97,  HNP97,  IEE97c,  Bal98,  IEE98b,  Sar98,  Pol99,  CooOla],  Model  and  simu- 
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lation  validation  “is  the  process  of  detennining  the  degree  to  which  a  model  is  an  accurate 
representation  of  the  real-world  from  the  perspective  of  the  intended  uses  of  the  model” 
[DoD94].  [Roa98]  views  the  validation  process  as  solving  the  right  equation.  The  De¬ 
partment’s  definition  of  validation  [DoD94]  includes  the  validation  of  conceptual  model, 
results,  and  data  in  the  V&V  process  supporting  multiple  Department  domain  requirements 
including  acquisition  programs  [CS97,  COHOO,  CoyOOb].  The  IEEE  defines  validation  is 
as  the  ’’confirmation  by  examination  and  provisions  of  objective  evidence  that  particular 
requirements  have  been  fulfilled”  [IEE98b]. 

There  are  many  validation  strategies,  with  several  noted  in  this  research.  [HNP97] 
identified  two  possible  strategies  to  validate  a  model:  the  axiomatic  approach  (e.g.,  theory- 
driven)  where  the  existence  of  a  set  of  assumptions  describe  the  fundamental  truths  of  the 
problem  domain,  and  the  model  proven  as  a  theorem;  and  the  empirical  approach  (e.g., 
data-driven)  in  which  the  model  performance  is  compared  to  expected  values  or  historical 
data.  Generalized  validation  (e.g.,  evaluation)  according  to  a  VV&A  taxonomy  developed 
by  [Dav92b]  would  include  empirical  and  theoretical  evaluation  methods  in  addition  to 
evaluation  by  other  methods.  Output  accuracy  methods  outlined  by  [Kil02]  includes 
benchmarking,  face  validation,  results  validation  and  sensitivity  analysis.  According  to 
[Kil02]  there  are  three  primary  ways  to  determine  output  accuracy:  another  M&S  or 
benchmarking,  subject  matter  expert  experience  employing  face  validation,  results  valida¬ 
tion  with  test  data,  with  all  methods  sometimes  supplemented  with  sensitivity  analysis.  In 
order  to  achieve  the  highest  degree  of  M&S  credibility,  [LKOO]  opine  that  the  “most  defini¬ 
tive  test  of  a  M&S  model’s  validity  is  to  establish  that  it’s  output  data  closely  resemble  the 
output  data  that  would  be  expected  from  the  actual  (proposed)  system”  [LKOO], 

The  degree  of  confidence  in  M&S  depends  heavily  on  M&S  anchoring  in  physical 
testing  [WelOO].  Although  tests  and  evaluations  may  develop  sufficient  confidence  levels 
that  the  model  is  valid  for  its  intended  purpose,  [Bal98]  cautions  that  complete  M&S  model 
testing  is  not  possible  and  that  successfully  testing  each  sub-model  (module)  does  not  im¬ 
ply  overall  model  credibility.  Furthermore,  [LKOO]  acknowledges  the  timing  and  relation¬ 
ship  of  validation  is  critical  for  establishing  and  improving  confidence  levels  that  “correct” 
results  are  available  for  credible  decisions.  Five  key  quality  factors  identified  by  [MLW01] 
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support  M&S  credibility:  capability,  software  accuracy,  data  accuracy,  results  accuracy  and 
usability. 

[BCE+99]  addresses  another  difficult  testing  challenge  of  striking  a  balance  be¬ 
tween  live  testing,  sound  analytical  methods  and  ground  testing.  M&S  is  extremely  chal¬ 
lenging  for  advanced  or  unprecedented  systems,  where  “substantial  design  deficiencies  or 
unanticipated  system  characteristics  requiring  correction  continue  to  be  revealed  during 
flight  tests  despite  the  sometimes  substantially  and  costly  applications  of  M&S.  ..even  those 
subjected  to  comprehensive  verification  and  validation”  [BCE+99]. 

Even  with  these  concerns,  there  are  still  differences  of  opinion  within  the  M&S 
community  today  in  the  requirement  for  V&V,  the  scope  of  the  effort,  the  high  cost,  how  it 
is  implemented,  specific  techniques,  and  detennining  “how  much  V&V  is  enough?”  In  ad¬ 
dition,  [Bal98]  suggests  that  some  tests  are  more  appropriate  to  evaluate  the  behavioral  ac¬ 
curacy  or  validity  of  the  model,  while  other  tests  better  determine  the  accuracy  and  verifi¬ 
cation  of  model  transformation  from  one  form  into  another.  Consequently,  the  Depart¬ 
ment’s  M&S  domain  “lacks  a  coherent  process  that  links  V&V  information  to  application- 
specific  requirements  for  M&S  credibility”[GMS+96]. 

3.  The  Department’s  Accreditation  Process 

The  final  action  of  the  Department’s  VV&A  process  is  the  decision-makers  accredi¬ 
tation8'1  [SS92,  CSC94a,  CSC94b,  JAS96,  San97a,  Sta97,  PacOla]  of  the  simulation.  M&S 
accreditation  is  “the  official  certification  that  a  model,  simulation  or  federation  of  simula¬ 
tions  is  acceptable  for  use  for  a  specific  purpose”,  implemented  by  the  Department 
[DoD95,  GMS+96]  to  provide  credibility  and  confidence  in  M&S  developed  for  another 
purpose  or  agency.  This  process  provides  the  decision-maker  with  an  understanding  of  the 
capabilities  and  limitations  (e.g.,  caveats)  of  the  simulation.  Accreditation  in  the  Depart¬ 
ment’s  M&S  domain  requires  confidence  in  the  V&V  process,  documentation,  data,  use 
history,  and  known  limitations  of  legacy  simulations,  especially  when  developed  for  other 
purposes,  or  maintained  by  different  organizations. 

83  Accreditation  in  terms  of  information  system  security  has  another  meaning:  The  formal  declaration  by  a  Designated  Approval  Author¬ 
ity  (DAA)  that  an  information  system  is  approved  to  operate  in  a  particular  security  mode  using  a  prescribed  set  of  safeguards  at  an  ac¬ 
ceptable  level  of  risk  (DITSCAP)  [MonOOb] 
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The  challenge  of  achieving  credibility  in  the  Department’s  M&S  was  succinctly 
noted  by  [San97a]  who  wrote  that  it  “is  not  confidence  in  M&S  we  seek,  but  confidence  in 
the  decisions  we  are  making”  [San97a].  [SS92]  propose  that  all  models  are  wrong  in  some 
aspects,  and  the  validation  process  identifies  how  and  when  the  model  is  erroneous,  while 
the  accreditation  process  is  the  “determination  that  the  decision  to  be  made  is  not  sensitive 
to  these  errors  and  limitations” [SS92].  As  emphasis  on  formal  VV&A  grew  within  the  De¬ 
partment,  two  kinds  of  accreditation  were  initially  proposed  according  to  [PacOlc]:  1)  class 
or  domain  accreditation,  as  a  form  of  blessing  of  the  simulation  for  a  kind  (e.g.,  class)  of 
application;  and  2)  specific  application  accreditation,  as  a  formal  certification  that  the  simu¬ 
lation  and  its  associated  inputs  is  appropriate  for  a  particular  use.  [PacOlc]  cautions  against 
the  temptation  to  use  class  accreditation,  noting  that  the  potential  exists  today  within  the 
Department’s  M&S  domain  to  “use. . .  almost  right,  but  not  quite  right  software”,  contrib¬ 
uted  to  the  uninsured  five  billion  dollar  loss  of  the  Ariane  5  booster  in  1996  [LLF+96]. 

Department  component  accreditation  responsibilities  include  the  accreditation  of 
forces  and  capabilities  employed  in  joint,  general,  and  common-use  M&S,  and  ensuring  the 
appropriate  representation  of  its  forces  and  capabilities  in  joint  and  common  use  M&S  de¬ 
velopment  [DoD94].  Simulation  credibility  defined  by  [Kil02]  includes  the  ability  of  the 
most  important  elements  to  meet  the  required  capability  and  intended  use.  Required  capa¬ 
bilities  explained  by  [Kil02]  include  descriptions  of  the  simulation’s  purpose,  represented 
model  entities  and  fidelity,  physical  environment  description,  element  relationships,  and 
assumptions  and  limitations.  Addressing  accreditation  of  federations,  [PacOla]  identifies 
the  compatibility  of  individual  federates  in  a  federation  as  a  key  element  in  validation  as¬ 
sessment  to  determine  if  the  federation  can  appropriately  support  the  intended  application. 

4.  The  Role  of  Data  in  the  VV&A  Process 

Data  is  critical  to  the  Department  goal  of  improving  credibility  and  confidence  in 
M&S  results,  and  plays  a  key  role  in  developing  simulation  credibility  and  underpinning 
user  confidence  in  the  results.  The  [RPGOO]  cites  four  basic  types  of  data  including:  1)  ref¬ 
erence  or  metadata,  2)  hard-wired  data  such  as  constants  and  set  parameters,  3)  instance 
data  for  input  and  output  functions,  and  4)  validation  data  comprised  of  actual  measure- 
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ments  or  real-world  facts  used  for  comparison  with  M&S  results  for  validation.  The  three 
basic  metadata  types:  administrative,  descriptive,  and  quality,  identify  databases  [Gor96], 
or  individual  data  elements  [RPGOO],  whereas,  metadata  quality  supports  the  quality  as¬ 
sessment  techniques,  results,  and  addresses  attributes  for  accuracy,  precision,  complete¬ 
ness,  portability,  consistency84,  currency  and  flexibility.  An  additional  fifth  type  of  data 
includes  exchange  metadata  for  federations.  Multiple  authors  including  [Atw91,  NMY+98, 
HSB98,  GozOO,  PFL+00,  DelOO,  PKL01,  SH01,  TolOl,  LWK+02]  address  data  verifica¬ 
tion,  validation  and  certification  issues. 

From  another  viewpoint,  [HNP97]  suggests  three  categories  of  data:  discrete  data, 
which  may  take  on  one  of  a  finite  set  of  values;  continuous  data,  which  may  assume  any 
value  in  an  interval  of  real  numbers;  and  categorical  data,  which  may  take  on  a  finite,  nor- 
mally  small  set  of  values.  Whatever  data  ontology  is  chosen,  authoritative  data  is  re¬ 
quired  for  developing  credible  representations  and  other  critical  activities  in  Department 
M&S.  However  the  [DoDOla]  cites  authoritative  data  as  “  the  single-most  pervasive  defi¬ 
ciency  area  identified  with  M&S  use”  [DoDOla]. 

Data  is  also  critical  to  the  Department’s  VV&A  process,  yet  widely  misunderstood 
and  sub-optimally  executed  in  the  Department.  [Sar98]  cited  historical  data  validation 
methods  such  as  rationalism  and  empiricism,  as  well  as  a  predictive  validity  approach  to 
improve  the  process.  Acquiring  authoritative,  certified  data  supporting  sufficient  M&S 
validation  processes  is  also  difficult,  and  exacerbated  by  the  existence  of  scientific  and 
technical  data  characterized  as  complex  data86.  Complex  data  categories  include  highly 
derived  data,  objects,  compositions  and  artifacts  of  legacy  systems  [DoD95]. 

A  major  Department  study  on  the  existing  data  verification,  validation  and  certifica¬ 
tion  (VV&C)  methodology,  [NMY+98],  concluded  that  data  quality  results  from  the  inte¬ 
gration  of  the  data  producer’s  V&V  process,  which  supports  the  user’s  data  V&V  efforts. 
Both  activities  are  components  of  the  M&S  V&V  process,  conducted  during  the  entire 

84 

Consistency  refers  to  data  that  is  maintained  free  from  variation  or  contradiction  [DoD98]. 

85  Authoritative  data  comes  from  authoritative  data  sources  whose  products  have  undergone  producer  verification,  validation,  and  certi¬ 
fication  activities  [DoD98]. 

Complex  data  cannot  be  characterized  as  a  single  concept  or  data  element  [DoD95] 

87  Highly  derived  data  includes  categories  such  a  probability  of  hit/kill.  Objects  employ  multiple  inheritance,  multiple  root  hierarchies 
and  polymorphic  attributes.  Composition  includes  images,  binary  large  objects  (blobs),  and  compound  documents.  Artifacts  are  nor¬ 
mally  attributed  to  data  from  legacy  systems  designed  to  reduce  storage  requirements  by  the  use  of  embedded  codes  or  logic  [DoD95]. 
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M&S  life  cycle,  with  data  quality  metadata  bridging  the  two  activities.  However, 
[NMY+98]  identified  the  lack  of  discrete,  formal  VV&C  standards  supported  by  data  qual¬ 
ity  metrics,  including  metadata  quality  metrics  as  a  systemic  Departmental  data  deficiency. 
Four  major  deficiencies  noted  by  the  [NMY+98]  study  included: 

•  Data  V&V  methods  lacked  consistency, 

•  No  linkage  between  the  development  of  data  products  and  user  V&V  requirements, 

•  No  central  repository  for  data  V&V  infonnation, 

•  Data  VV&C  not  synchronized  with  the  M&S  VV&A  effort  [NMY+98], 

The  concept  that  data  V&V  should  be  an  integral  part  of  the  M&S  life-cycle  proc¬ 
ess  [IEE96,  IEE97b]  as  opposed  to  a  separate  distinct  procedure  has  been  emphasized  by 
recent  research  [Sar98,  NMY+98],  and  reflected  in  Figure  3-3.  This  process  is  a  variant  of 
the  earlier  recommended  procedure  in  the  [DoD95]  guidelines  defining  a  separate  process 
for  verification,  validation  and  certification  (VV&C)  of  data  by  the  producer  and  the  user. 
The  separate  VV&A  and  VV&C  processes  raised  the  probability  that  validation  of  the  of 
data  for  the  intended  use  was  more  implied  than  the  result  of  an  explicit,  defined,  repeat- 
able  process. 

Several  methodologies  exist  to  improve  data  quality.  Data  quality  [DoD98]  is  the 
correctness,  timeliness,  accuracy,  completeness,  relevance,  and  accessibility  characteristics, 
which  make  data  appropriate  for  use.88  Validated  data  for  rapidly  evolving  M&S  is  a  major 
challenge,  but  is  essential  for  M&S  tools  if  they  are  to  achieve  their  full  potential.  Data 
accuracy  cited  by  [Kil02]  addresses  the  appropriateness  of  the  data,  including  resolution; 
the  quality  of  the  data  established  by  the  data  producer  and  user;  and  the  correctness  of  data 
transfonnations.  [Cau95]  proposes  a  reduced  order  metamodel  validation  methodology  to 
compare  M&S  results  with  real-world  data,  if  available.  Suggested  improvements  in  the 
data  validation  process  supported  by  [HHOOa,  HSOOb]  address  data  interchange  formats 
(DIF),  while  [HSB98]  recommend  a  CMMS  data  dictionary  for  improved  interoperability. 
[DoDOla]  adds  the  requirement  for  new  V&V  methodologies  and  improved  technologies, 
including  software  quality  metrics  detailed  by  [Kan03]  to  enhance  the  software  quality 
V&V  practices  within  the  Department. 

88  Quality  statements  are  needed  for  source,  accuracy  (positional  and  attribute),  currency,  logical  consistency,  completeness  (feature  and 
attribute),  security  classification,  and  releasibility  [DoD95] 
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Figure  3-3.  Data  VV&A  &  Legacy  M&S  (From  [RPGOO]) 

[Dav92b]  reinforced  the  need  for  improved  life-cycle  VV&A  practices  with  rec¬ 
ommendations  to  develop  positive  incentives  including  a  vision  for  VV&A,  and  the  devel¬ 
opment  of  consistent  policy  and  procedures  across  the  Department.  [Dav92b]  further  rec¬ 
ommended  early  successful  implementations,  tailoring  of  VV&A  to  the  corporate  culture, 
establishing  independent  reviews,  providing  long  and  short-range  planning  processes,  and 
understanding  project/program  concerns  with  VV&A.  [JorOl]  notes  that  data  V&V  should 
be  performed  as  an  integral  part  of  the  V&V  process,  stating  that  “algorithms  only  work  if 

the  required  data  is  available  and  is  accurate  for  the  aggregation  portrayed”  [JorOl]. 
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Establishing  credibility  and  consistency  according  to  [HBM96]  requires  the  publi¬ 
cation  of  the  approach,  data,  and  tests  be  for  peer  review  to  improve  the  current  situation  of 
too  many  redundant  databases,  too  little  testing,  too  few  reviews  of  data  sources,  and  insuf¬ 
ficient  analysis  of  embedded  assumptions.  Recent  works  by  [MSOOl]  on  missing  data 
techniques  and  [SC02]  on  sparse  data  method  have  addressed  some  of  the  issues  of  sparse 
or  missing  historical  data.  However,  at  the  present  time,  the  Department’s  objective  for  an 
over-arching  coherent  program  for  affordable,  timely,  verified,  validated,  and  authoritative 
data  on  demand  is  still  more  a  goal  than  a  reality  [CraOlb]. 

5.  Verification  and  Validation  (V&V)  Techniques 

The  Department’s  V&V  process  for  M&S  is  the  approved  foundation  for  achieving 
quality  and  developing  M&S  credibility.  A  current  Department  M&S  objective  includes 
maintaining  and  enhancing  the  V&V  program  for  models,  simulations  and  data  to  provide 
users  with  confidence  in  the  results  and  knowledge  of  the  limitations.  The  Department  has 
over  seventy- five  software  engineering  and  model  specific  techniques  for  performing  V&V 
exist  today,  with  methods  ranging  from  infonnal  to  formal  methodologies  [GMS+96, 

Pol98,  RPGOO].  Figure  3-4  identifies  the  infonnal,  static,  dynamic  and  formal  V&V  tech¬ 
niques. 

The  mathematical  and  logical  fonnalism  increases  from  the  infonnal  techniques  on 
the  left  to  the  formal  methods  on  the  right,  as  does  the  complexity  of  the  techniques.  In¬ 
fonnal  V&V  techniques  noted  in  Figure  3-4  lack  rigorous  mathematical  formalism.  These 
tools  and  methods  employing  human  reasoning  and  subjective  techniques  may  be  very  ef¬ 
fective  if  applied  using  well-structured  approaches,  fonnal  guidelines  and  mature,  disci¬ 
plined  processes  [GMS+96].  Informal  V&V  techniques  are  among  the  most  common,  fre¬ 
quently  used  approaches  in  the  Department  [RPGOO], 

Static  V&V  techniques  cited  in  Figure  3-4  provide  insights  into  the  models  struc¬ 
ture,  modeling  techniques,  data,  control  flow  and  syntax.  Different  V&V  approaches  ad¬ 
dressed  by  [Pac02b]  depend  on  whether  the  M&S  source  code  is  available.  V&V  agents 
employ  these  techniques  to  review  the  accuracy  of  the  model  design  and  source  code. 

These  techniques  are  widely  used  and  do  not  require  model  execution  runs.  Quality  auto- 
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mated  support  tool  requirements  identified  by  [AWQ+93,  ACC+94,  MBH99,  GFM01]  are 
also  necessary  if  the  Department  is  to  realize  the  goals  of  improved  results  from  VV&A 
activities.  Few  current  automated  tools  are  available  to  assist  in  this  process,  except  the 
language  compiler  or  interpreter  [GMS+96,  RPGOO]. 
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Figure  3-4.  V&V  Techniques  (After  [GMS+96,  RPGOO]) 

Dynamic  V&V  Techniques  provided  in  Figure  3-4  evaluate  execution  behavior  of 
the  model  during  run-time.  One  technique  inserts  probes  or  stubs  into  the  executable  code 
to  collect  infonnation  during  the  running  of  the  M&S.  This  method  normally  employs 
three  steps:  inserting  the  probe  or  stub  instrumentation,  executing  and  collecting  the  re¬ 
quired  data,  and  analyzing  the  output  file  for  discrepancies  or  faults.  Dynamic  methods 
and  statistical  techniques  have  been  addressed  in  a  significant  body  of  Department  V&V 
guidelines  [GMS+96,  RPGOO],  [RSH90]  submit  a  propagative  approach  to  sensitivity 
analysis  in  large-scale  M&S  where  the  sheer  numbers  of  parameters  make  it  impractical  to 
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test  more  than  a  small  fraction  of  relevant  cases,  a  common  challenge  in  all  large  software¬ 
intensive  systems. 

Figure  3-4  lists  a  number  of  various  statistical  V&V  techniques  available  to  validate 
models.  The  best  environments  for  these  techniques  include  completely  observable  models 
and  provide  for  the  collection  of  all  data  for  validation.  In  this  category,  [NF67]  advanced 
the  technique  of  Analysis  of  Variance,  while,  [MM98]  suggest  peer-reviewed  statistical 
techniques  for  effectively  managing  model  fidelity.  In  addition  [MM98]  contend  that  De¬ 
partment  models  often  comprise  a  set  of  sub-model  requiring  additional  integration  into  a 
coherent  representation  of  a  mission  space. 

The  literature  also  reveals  an  abundance  of  additional  V&V  techniques  and  proce¬ 
dures.  [Ham96,  HNP97,  Bal98,  LKOO]  discussed  simultaneous  confidence  levels,  confi¬ 
dence  levels  and  model  range  of  accuracy  to  address  operational  validity.  [Ham96, 

HNP97,  LKOO]  also  explained  several  statistical  procedures  (inspection,  time-series  ap¬ 
proaches)  for  comparing  real-world  data  to  M&S  output.  [ROG98]  proposed  a  multistage 
validation  framework  consisting  of  conceptual  and  operational  validation  employing  two 
statistical  methods,  a  two-sample  t  test  and  a  two-dimensioned,  two-sample  Kohnogorov- 
Smimov  test  [HNP97],  concluding  that  a  model  valid  for  one  level  of  detail  may  be  invalid 
for  another  use.  [Ham96,  HNP97]  suggests  the  validation  process  is  ambiguous,  complex, 
and  difficult,  subject  to  undesirable  subjective  approaches,  in  part  because  the  first  part  of 
the  validation  process  builds  upon  a  credible  validation  of  the  conceptual  model,  itself  an 
abstraction. 

In  addition,  [FriOl]  explains  the  application  of  Fisher’s  Combined  Probability  Test 
and  Goodness-of-Fit  Tests  to  support  fonnal  statistical  comparisons,  even  when  “the  data 
are  sparsely  distributed  across  a  highly  dimensional  factor  space”  [FriOl].  [HalOla]  sug¬ 
gests  that  the  breakdown  of  the  model  into  testable  functional  elements  supports  validation 
data  requirements  and  makes  the  point  that  data  “obtained  from  testing  in  this  manner  is  at 
a  low  enough  level  in  the  model  to  support  statistical  comparisons  between  model  predic¬ 
tions  and  test  results”  [HalOla].  Classical  statistical  techniques  [LKOO]  that  work  well  with 
ideally  distributed,  independent  observations,  may  prove  difficult  to  apply  correctly  with  all 
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M&S  situations  since  both  real-world  systems  and  M&S  output  produce  non-stationary  dis¬ 
tributions  and  auto-correlated  observations  [RPGOO]. 

Disciplined  verification,  validation,  and  accreditation  techniques  for  M&S  and  data, 
support  interoperability  for  new  large-scale,  complex  simulations,  and  federations.  A  sur¬ 
vey  of  current  M&S  model  VV&T  techniques  [Bal98]  apply  throughout  the  system  life  cy¬ 
cle.  Several  additional  methods  and  techniques  to  improve  validity  prescribed  by  [HNP97, 
LKOO]  include:  collect  high-quality  data,  maintain  an  assumption  documents,  conduct 
structured  walk-through,  validate  model  using  quantitative  techniques,  validate  the  output, 
and  use  animation  to  improve  user  understanding  of  the  system.  [BML02]  further  suggests 
a  theoretical  framework  for  determining  the  confidence  in  a  simulation  based  on  the  credi¬ 
bility  of  three  factors:  the  model  and  its  embedded  data,  runtime  data,  and  the  operation  or 
application  of  the  model. 

[Sar98]  identified  operational  validity  techniques  for  M&S  test  programs,  which  in¬ 
clude  event  validity,  comparison  to  other  models,  degenerate  tests,  extreme  condition  tests, 
fixed  values,  face  validity,  parameter  variability-sensitivity  analysis,  traces  and  Turing 
Tests.  [Sar98]  also  proposes  hypothesis  tests,  which  support  a  comparison  of  means,  vari¬ 
ances,  distributions,  and  time  series  of  the  output  variables  to  determine  if  the  model’s  out¬ 
put  behavior  has  an  acceptable  range  of  accuracy.  However,  the  current  state  of  M&S 
software  development  maturity  largely  lacks  objective  V&V  processes  with  quantitative 
methodologies  supporting  the  sufficiency  of  a  model  for  a  specific  purpose. 

The  highest  possible  confidence-levels  achievable  in  the  V&V  process  include  the 
fonnal  V&V  techniques  noted  in  Figure  3-4.  Formal  methods  are  also  the  most  complex, 
least  understood,  and  least  implemented  V&V  techniques  in  the  Department  [GMS+96]. 
The  scope  of  formal  method  techniques,  processes,  and  procedures  include  [LG93,  LGB94, 
FMG94,  GMS+96,  LG97,  Ber98a,  Ber98b,  LBOO,  RPGOO,  MLJ01,  MJL01,  POPOO, 

BC02],  including  the  changes  of  formal  methods  [Rob98]  employ  mathematical  proofs  for 
correctness,  and  require  significant  effort.  Formal  methods  applied  early  in  the  develop¬ 
ment  process,  normally  achieve  the  best  results.  However,  formal  methods  applied  to  the 
most  critical  components,  may  prove  effective  at  any  time  during  the  development  process 
[Mic03], 
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[NogOO]  developed  a  formal  method  to  automatically  assess  risk  and  the  duration  of 
a  software  development  project  supported  by  evolutionary  software  processes.  For  large- 
scale  software  engineering  projects,  [BL91]  proposed  a  formal  methods  mathematical  ap¬ 
proach  as  a  basis  for  precise  communication  and  the  foundation  to  achieve  a  high  degree  of 
automation  directly  supporting  increased  confidence  in  the  software  quality.  NASA  also 
realized  the  increasing  complexity  and  criticality  of  software  applications  required  an  em¬ 
phasis  in  formal  methods.  Major  factors  identified  by  [NAS97,  CVL+97,  CKM+98]  for 
increased  use  of  fonnal  techniques  included  the  following: 

•  Fault-protection  and  safety  functions  required  software  features  beyond  the  hard¬ 
ware  level  to  detect  and  isolate  failures,  and  initiate  recovery  procedures, 

•  Software-failure  characteristics  were  different  than  those  found  in  hardware, 

•  Complex  systems  placed  ever-growing  demands  on  verification, 

•  Disciplined  software  engineering  organizations  had  developed  fine-tuned  verifica¬ 
tion  processes,  but  defects  still  remained  in  the  product, 

•  Few  existing  techniques  were  rigorous  enough  for  improving  the  quality  of  products 
during  the  early  life-cycle  phases  of  requirements  generation  and  design 
[CKM+98], 

The  overlapping  tools,  techniques,  procedures  and  tenns  of  reference  between  the 
M&S  domain  and  emerging  software  engineering/development  domains  continually  mani¬ 
fest  themselves  in  the  literature.  [MLU+00]  propose  an  incremental  qualification  approach 
to  M&S  VV&A,  noting  that  M&S  “is  subject  to  software  development. .  .as  well  as  system 
engineering  practices.  [MLU+00]”  [ARS+96]  also  identified  V&V  as  a  system  engineer¬ 
ing  discipline,  which  must  be  included  in  the  domain  engineering  process  to  improve  the 
level  of  confidence  in  critical  software,  specifically  safety  and  mission-critical  software. 

The  Joint  Accreditation  Support  Activity  (JASA)  developed  and  continuously  re¬ 
fined  the  Susceptibility  Model  Assessment  and  Range  Test  (SMART)  methodology 
[JAS97a,  JAS97b].  The  SMART  methodology  involves  a  four-phase  approach  to  VV&A 
for  acquisition  M&S  including:  incremental  reviews,  analysis,  evaluation,  testing,  and 
documentation  of  M&S  and  associated  data  to  support  the  verification  and  validation  proc¬ 
ess  detailed  by  [CSC94a,  CSC94b].  With  over  ten  years  of  VV&A  related-experience  in  a 
wide  range  of  M&S  programs,  the  JASA  identified  eight  major  obstacles  to  VV&A  success 
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similar  to  the  systemic  issues  adversely  impacting  Department  M&S  credibility  identified 
earlier  by  [PAD78,  PAD79a,  PAD79b,  PAD80]: 

•  Lack  of  cooperation  from  the  M&S  developer  based  on  concerns  that  identification 
of  errors  could  adversely  impact  future  business,  intellectual  property  [IEE99b]  is¬ 
sues,  and  failure  to  specify  V&V  in  contract  tasks, 

•  Failures  to  follow  disciplined  processes,  such  as  configuration  management,  and 
develop  quality  documentation, 

•  Failure  to  define  intended  use  and  M&S  requirements,  including  functionality  and 
fidelity, 

•  Failure  to  define  accreditation  information  requirements, 

•  Lack  of  or  mismatch  of  skills, 

•  Insufficient  priority  applied  to  VV&A  efforts, 

•  Lack  of  interpersonal  skills  demonstrated  by  VV&A  team  members, 

•  Lack  of  perseverance  [JAS97a,  JAS97b]. 

The  private  sector  contractors  supporting  Department  M&S  initiatives  experienced 
the  same  systemic  issues  establishing  M&S  credibility  and  maintaining  users  confidence  in 
M&S  results.  A  1993  Department  audit  report  [RSV+93]  of  a  sample  of  defense  contractor 
M&S  programs  identified  costly  duplication,  redundancy  and  unnecessary  proliferation  of 
existing  M&S  capabilities  with  the  following  quality  indicators: 

•  Only  twenty  percent  of  the  defense  contractor  M&S  had  completed  a  formal  VV&A 
program, 

•  Five  percent  of  defense  contractor  M&S  had  adequate  configuration  management 
controls  in  place, 

•  Thirty  five  percent  of  the  defense  contractor  M&S  had  adequate  documentation, 

•  The  findings  indicated  “a  substantial  lack  of  effective  management  control  over  the 
development  and  use  of  M&S  by  the  government  contractor  community” 

[RSV+93], 

6.  Shortfalls  of  the  Department’s  VV&A  Process 

There  are  also  many  V&V  shortfalls,  limitations,  and  critiques  of  the  Department’s 
V&V  process  cited  in  the  literature  including  [Che87,  Dav92b,  LA92,  Met92,  Pay96, 

TP96,  CVL+97,  KP97,  HM98,  ML97,  Mue97a,  NAS97,  JDD98,  Lew98,  NM98,  ROG98, 
Pac99b,  MLU+00,  TRWOO,  CraOlb,  HNK+01,  JorOl,  MT98,  HS02,  Kil02,  Pac02b]. 

There  are  also  limits  to  the  confidence  established  by  validation  with  well  known  validation 
limits  [Kil02]  including  the  limited  scope  of  validation  tests,  costly  validation  data  which  is 
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difficult  to  obtain,  and  limited  tools  [Lew94,  Bar98,  DMSOOb].  In  addition,  conventional 
methods  and  techniques  may  not  be  adequate  to  validate  some  highly  aggregated  M&S 
(e.g.  mission  and  campaign-level  models)  [Kil02] . 

High-fidelity  models,  HLA  federations  [HLA98],  and  legacy  M&S  issues  [LBS01, 
ZB02]  pose  additional,  significant  challenges  for  the  proper  execution  of  V&V.  Effective 
and  efficient  validation  of  all  potential  logic  paths  in  the  M&S  is  probably  technically  and 
economically  infeasible,  since  the  number  of  possible  system  states  in  a  large  simulation  is 
very  large.  The  number  of  states  grows  exponentially  with  regard  to  the  number  of  degrees 
of  freedom  in  the  model  simulation  or  software  [Mic03].  [Pac02b]  suggests  that  the  utility 
of  M&S  is  limited  if  the  M&S  validity  and  fidelity  cannot  be  adequately  quantified.  This 
includes  testing  and  test  support  V&V  issues  identified  by  [Kri88,  HN97,  Che98,  Kec99, 
Obr99,  WhiOO,  LLC+02]. 

A  finding  of  the  1999  Defense  Science  Board  on  Test  and  Evaluation  [HAC+99] 
concluded  that  “Verification,  Validation  and  Accreditation  as  presently  practiced  with  re¬ 
spect  to  M&S  techniques  in  general  is  not  sufficiently  disciplined  to  inspire  confidence  in 
their  use  in  the  T&E  process”  [HAC+99].  [HAC+99]  further  proposed  better  coordinated 
and  more  disciplined  M&S  software  development  processes  supported  by  improved 
VV&A  to  address  the  following  findings  on  M&S  use  in  the  test  and  evaluation: 

•  M&S  is  effective  and  cost  effective  in  many  clearly  defined  applications  where  both 
the  expertise  and  data  are  available  to  support  a  common  mission, 

•  Large-scale  constructive  force-on-force  combat  M&S  tend  to  be  the  least  useful 
when  the  inputs  are  largely  unknown  or  uncertain, 

•  The  current  Department  M&S  investment  emphasizes  model  architectures,  inter¬ 
faces,  graphical  displays,  and  programming  in  lieu  of  conceptual  model  develop¬ 
ment  and  basic  data  collection, 

•  No  single,  authoritative  catalog  or  list  of  available  M&S  and  quality  attributes  ex¬ 
ists, 

•  Open  air  testing  is  not  feasible  for  some  weapon  systems,  creating  a  dependency  of 
some  degree  on  M&S  for  evaluations  of  weapon  systems, 

•  M&S  has  potential  in  the  test  design  and  planning  process,  and  the  reliability,  avail¬ 
ability  and  maintainability  (RAM)  areas, 

•  Cost  and  benefits  are  difficult  to  measure,  up-front  funding  is  often  lacking,  and 
there  is  a  concern  that  the  Department  cannot  maintain  the  talent  necessary  to  im¬ 
prove  M&S  capability  [HAC+99]. 
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For  all  the  potential  benefits  associated  with  M&S,  the  actual  achievements  fall 
short  of  predicted  performance  objectives,  simulations  lack  widespread  credibility  to  instill 
user  confidence,  and  the  community  still  requires  an  acceptable  method  for  determining  the 
return  on  investment  for  M&S.  [PHP+96]  identified  challenging,  systemic,  and  recurring 
issues  of  credibility  in  the  tools  and  serious  doubts  about  the  confidence  of  M&S  results 
with  the  “underlying  issues  of  the  credibility  of  the  tools  being  a  major  constraint” 
[PHP+96],  Only  forty- five  percent  (20  of  44  systems)  of  the  model  summaries  reviewed  in 
a  1998  Department  medical  M&S  catalog  [Gau98]  revealed  any  V&V  activities.  The  in¬ 
ability  to  adequately  quantify  M&S  fidelity  and  validity  limits  overall  M&S  utility. 

In  a  commentary  of  the  current  state  of  M&S  [Whi99]  suggests  that  VV&A  has 
been  an  after-thought,  a  topic  of  discussion,  but  not  properly  implemented.  These  limita¬ 
tions  acknowledges  [PacOlb]  make  it  difficult  to  establish  confidence  in  M&S  results,  iden¬ 
tify  the  risks  associated  with  decisions  based  upon  M&S  results,  or  detennine  the  appropri¬ 
ate  resource  levels  to  increase  confidence  levels  in  the  M&S.  More  importantly,  these  ex¬ 
amples  are  just  instances  of  the  many  issues  identified  by  [DOTOla]  where  the  inadequate 
implementation  of  VV&A  policies  may  result  in  the  use  of  non-credible  M&S  to  support 
critical  Department  program  acquisition  decisions. 

7.  Secondary  Credibility  Contributing  Factors 

In  addition  to  the  primary  VV&A  procedures,  several  other  mitigating  factors  sur¬ 
faced  often  in  the  literature  and  appeared  to  have  an  important  secondary  affect  on  deci¬ 
sions  to  initiate  or  conduct  a  VV&A  program.  We  call  these  secondary  credibility- 
contributing  factors,  and  these  factors  include  reuse,  documentation  or  lack  of  documenta¬ 
tion,  risk  management,  cost  of  V&V,  and  the  affect  of  V&V  on  a  simulations  return  on  in¬ 
vestment  (ROI).  Employed  consistently,  the  secondary  credibility-contributing  factors 
support  credibility,  may  reduce  overall  life  cycle  cost,  and  identify  a  return  on  investment. 
Unfortunately,  the  Department’s  complex  organizational  dynamics,  and  complicated  acqui¬ 
sition  procedures  [Wol02a,  Wol02b]  minimized  the  potential  contributions  of  these  factors. 
This  list  of  credibility-contributing  factors  is  not  all-inclusive,  however,  we  selected  them 
for  their  potential  influence  on  the  development  of  software-intensive  simulation  systems. 
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a. 


Reuse 


Software  reuse  [IEE99a,  IEE99b]  is  the  “practice  of  using  existing  software 
components89  to  develop  new  applications’’  [DCC+93,  KDM+94,  IEE99a,  IEE99b,  Jaz02]. 
Software  reuse  has  the  potential  to  reduce  development  costs,  reduce  time  to  market,  im¬ 
prove  simulation  quality  and  support  the  V&V  process,  according  to  [Som95]  and  [Pres97] 
who  identify  and  explain  the  salient  considerations  supporting  successful  software  reuse. 

In  1991,  the  Department  established  the  Software  Reuse  Initiative  (SRI)  to  collaboratively 
and  cooperatively  advance  the  reuse  strategy  with  other  government  agencies,  such  as 
NASA,  industry,  and  academia  to  make  software  reuse  effective  for  the  Department90 
[RTC94].  Software  reuse  faced  many  early  challenges,  which  included  organizational  ac¬ 
ceptance,  management  support,  legal  issues,  intellectual  property  rights,  liability  concerns 
and  changes  to  existing  acquisition  policies  [DCC+93,  IEE99b]. 

Reuse  evolved  according  to  [MooOl]  and  the  current  focus  is  now  on  “reus¬ 
ing  knowledge  itself,  not  just  life-cycle  artifacts  such  as  specifications,  code,  and  test  data” 
[MooOl].  Departmental  M&S  policies  also  support  reuse  noting,  “When  possible,  existing 
information  systems  shall  be  used  or  modified  rather  than  creating  new  systems”  [DoD94]. 
[Pac02a]  suggests  that  as  the  ability  to  automatically  generate  code  from  specifications  im¬ 
proves,  the  focus  of  reuse  efforts  should  be  in  the  earlier  activities  of  the  software  life  cy¬ 
cle,  which  precede  coding,  and  smaller  components  may  be  easier  to  reuse.  [TJOO,  MooOl] 
reinforce  this  concept  and  propose  domain  engineering  and  analysis  as  the  key  to  reusing 
knowledge,  capturing  common  elements  across  the  domain,  to  further  additional  domain 
development. 

In  the  M&S  domain,  explicit,  well-documented  conceptual  models  would  be 
prime  candidates  for  reuse,  based  on  both  definitions  of  the  term  reuse.  Unfortunately,  as 
[DoDOla]  notes,  past  investments  in  M&S  have  not  resulted  in  the  Department’s  capability 
to  identify  algorithms,  data  sets,  models  or  simulations  for  reuse.  Reuse  of  conceptual 


89 

Reusable  software  components  can  be  executable  programs,  code  segments,  documentation,  requirements,  design  and  architectures, 
test  data  and  test  plans,  or  software  tools.  They  may  also  be  knowledge  and  information  needed  to  develop,  use,  or  maintain  the  compo¬ 
nent.  [DCC+93] 

90  The  DoD  effort  included  the  ARPA  Software  Technology  for  Adaptable,  Reliable  Systems  (STARS)  effort,  the  Army  Software  Reuse 
Initiative,  the  Navy  Software  Reuse  Initiative,  the  Air  Force  Central  Archive  for  Reusable  Defense  Software  (CARDS),  and  the  Defense 
Information  Systems  (DISA)  Software  Reuse  Program  (SRP).  [RTC94] 
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models  in  the  Department  is  inconsistent  since,  “Many  Defense  community  simulations”, 
according  to  [Pac02a],  “lack  explicit  descriptions  of  the  conceptual  models  which  are  im¬ 
plemented  in  them”  [Pac02a],  This  is  a  central  theme  and  key  driver  of  this  dissertation. 

Recent  reuse  techniques  explored  knowledge-based  tutoring  systems  applied 
to  software  reuse  [NLR99],  computer-aided  tools  employing  the  relational  hydrograph 
model  and  computer-aided  software  evolution  system  [Har99a],  a  software  reuse  reference 
model  [RN99],  the  extension  of  existing  reuse  schemes  with  systemic  software  reuse  meth¬ 
odologies  and  reuse  success  factors  (e.g.,  planning,  management  support,  process  control, 
architecture,  and  domain  focus)  [RDK+99],  and  the  employment  of  adapters  to  encapsu¬ 
late,  and  connect  multi-use  component  interactions  [RNJ99].  Other  more  abstract  ap¬ 
proaches  identified  architecture  reuse  possibilities  based  on  component-based  reuse  of  a 
domain-specific  software  architecture  [HBL97],  architectural  mismatches  [GA095],  and 
the  architectural  style  approach  [MG96]  for  classifying,  storing,  retrieving  reusable  archi¬ 
tectural  design  components. 

[HBL97]  cites  software  component  reuse  [SLM91,  LBG+96]  and  rapidly 
changing  requirements  [RL95,  LBL+97]  as  the  two  major  problems  confronting  large, 
complex  software  development,  and  suggests  software  evolution  fonnalization  as  the  best 
approach  to  understand  the  development  and  evolution  process  [BL94,  DLB94,  LG97].  A 
recent  survey  by  [MET02]  of  large  and  small  Europeans  companies  employing  reuse,  con¬ 
firmed  these  problems  and  found  that  approximately  thirty- three  percent  of  reuse  projects 
in  the  survey  population  failed.  Although  the  sampled  companies  belonged  in  different 
business  domains,  [MET02]  found  successful  reuse  projects  shared  good  application  com¬ 
monality,  high  process  maturity  levels,  and  use  of  standard  procedural  or  object-oriented 
development  methodologies.  Successful  reuse  projects  also  shared  the  following  attributes: 

•  High  commonality  among  applications, 

•  Reuse  is  basically  a  technology  transfer  endeavor  and  requires  management  com¬ 
mitment, 

•  The  approach  to  reuse  may  be  standard,  but  the  deployment  approach  produced  the 
best  results  when  tailored  to  the  context  of  the  organization  [MET02], 

• 

[RDB+00]  suggests  the  current  open  source  software  movement,  including 

Linux,  is  a  logical  extension  of  both  the  traditional  reuse  methodologies  and  other  previous 

68 


open  source  standards  including  UNIX,  Sendmail,  Simple  Mail  Transfer  Protocol  (SMTP) 
for  email,  and  the  gee  compiler.  However,  since  reuse  is  not  part  of  software  engineering 
curriculums,  [MET02]  “forecast  there  will  be  a  long  delay  before  sensible  advances  are 
made”  [MET02].  Progress  has  been  noted  in  the  reuse  of  communication  protocols.  When 
problems  occur,  the  cause  may  be  the  mixing  and  matching  of  protocols,  and  using  them 
for  an  unintended  purpose  (e.g.,  at  the  MAC  layer  rather  than  the  session  layer)  [Mic03], 

b.  Documentation 

Software  documentation  is  an  important  software  artifact.  Complexity  of 
large  software  development  projects  drives  the  need  for  documentation  to  provide  commu¬ 
nications  between  project  members,  customers,  and  management.  [STS93a,  STS93b, 
AS94]  address  documentation  principles,  methods,  products,  tools  and  management  tech¬ 
niques  supporting  improved  quality  and  productivity. 

Providing  users  with  knowledge  of  M&S  limitations  and  developing  confi¬ 
dence  in  simulation  results  is  a  major  Departmental  M&S  objective.  [Sar98]  supports 
model  V&V  documentation  and  a  detailed  basis-of-confidence  document.  [LKOO]  pro¬ 
poses  maintaining  an  assumption  document  to  support  the  conceptual  model  development. 
This  method  also  has  potential  to  support  a  model  basis  of  confidence  document. 

However,  documentation  is  usually  a  low  priority  for  the  software  devel¬ 
oper.  In  addition,  [Kil02]  confirms  that  M&S  capability91  requirements,  seldom  under¬ 
stood  by  the  user  and  rarely  documented  with  useful  detail  beyond  top-level  users  manuals 
continue  to  exist  as  major  simulation  development  shortfalls.  As  a  result,  documentation 
for  Departmental  M&S  systems  is  often  incomplete,  missing,  outdated,  or  lacking  detail 
[DoDOla], 


Simulation  capability  descriptions  should  at  a  minimum  include  the  purpose,  modeled  elements,  environment,  relationship,  assump¬ 
tions,  and  limitations  [Kil02a], 
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c.  Risk  Management 

There  are  many  widely  used  definitions  for  the  concept  of  risk,  including  the 
IEEE’s  definition  [IEE01]92.  In  similar  views,  [BCC+00]  defines  risk  as  the  probability  of 
loss,  while  [HHG+02]  views  risk  as  the  probability  that  a  future  event  may  or  may  not  oc¬ 
cur,  and  the  consequences.  In  this  research  we  employ  the  risk  management  structure  and 
definitions  from  the  Risk  Management  Process  Model  [AndOl].  Risk  is: 

A  measure  of  the  potential  inability  to  achieve  overall  program  objectives 
within  defined  program  cost,  schedule,  and  technical  constraints  and  has  two 
components:  (1)  the  probability/likelihood  of  failing  to  achieve  a  particular 
outcome,  and  (2)  the  consequence/impacts  of  failing  to  achieve  that  outcome 
[AndOl], 

The  Risk  Management  Process  Model  [AndOl]  includes  the  following  events,  processes  or 
procedures,  varying  with  the  phases  of  the  system  acquisition  and  system  definition,  and 
integrated  into  the  program  management  function:  risk  events,  risk  management,  risk  plan¬ 
ning,  risk  assessment,  risk  handling,  risk  monitoring,  and  risk  documentation. 

[HHG+02]  also  cites  heuristic  approaches  to  risk  management  including  the 
probability  of  the  future  event  occurring,  the  consequence  of  occurrence,  and  identifies 
guides,  checklists  and  standard  operating  procedures  to  reduce  or  mitigate  risk.  [Boe91, 
Pfl98,  Gal99,  BCC+00,  RPGOO,  AndOl,  HDK+02,  HHG+02]  identified  software  risk  and 
risk  management93  approaches  in  software-intensive  information  systems  including:  guide¬ 
lines,  checklists,  and  taxonomies.  These  methods  also  include  the  probabilistic  techniques 
for  ex  post  facto  analysis  or  software  reliability  and  formal  statistical  approaches  detailed 
by  [RPGOO]. 

In  decision-making  there  are  two  basic  types  of  risk:  rejecting  correct  evi¬ 
dence  or  accepting  incorrect  evidence.  In  M&S  software  development  the  risks  of  accept¬ 
ing  incorrect  evidence  are  characterized  in  the  [RPGOO]  as  either  development  or  opera¬ 
tional  risks.  The  Department’s  VV&A  process,  designed  to  reduce  development  risk  by 
reducing  errors  and  defects  early  in  the  development  process,  theoretically  reduces  the 

92  ...  . 

Risk  is  the  likelihood  of  an  event,  hazard,  threat,  or  situation  occurring  and  its  undesirable  consequences;  a  potential  problem  [IEE01]. 

93  .  .  .... 

Software  risk  management  is  a  software  engineering  practice  with  processes,  methods,  and  tools  for  managing  risk  in  a  project.  It 

provides  a  disciplined  environment  for  pro-active  decision-making  to  assess  continuously  what  can  go  wrong,  determine  important  risks, 
and  implement  actions  to  deal  with  the  risks  [BCC+00]. 
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number  of  operational  risks  of  incorrect  M&S  outputs.  However,  [HHG+02]  considers 
software  integration  risk  as  the  largest  unknown  “unknown”  risk  based  on  the  current  soft¬ 
ware  development  paradigm  of  integrating  varied  and  unstable  software  components  such 
as  database  management  systems,  search  engines  and  web  browsers,  which  are  constantly 
evolving  to  remain  competitive  in  the  market  place. 

The  risk  of  failure  normally  occurs  in  three  ways  and  with  different  combi¬ 
nations  of  cost,  performance,  and  schedule.  These  occurrences  further  decompose  into  the 
following  risk  areas  of  technical,  supportability,  programmatic,  cost,  and  schedule  risks. 
[BCC+00]  identify  several  common  risk  factors  including:  management,  ignoring  risks, 
discipline,  training,  fallacy  of  easy  solutions,  work  plans,  schedule,  delivery,  constraints, 
customer,  methods,  and  tool  selection.  Common  risk-reduction  methods,  tools  and  proc¬ 
esses  exist  today  to  including  project  management  tools  such  as  PERT/CPM/Gantt  charts 
and  software  estimation  methods  incorporated  in  software  estimation  tools  [GAB97]. 

Multiple  organizations  in  the  private,  government,  and  academia  sectors 
continue  to  develop  advanced  risk  management  techniques,  procedures,  and  automated 
tools,  including  the  International  Council  on  Systems  Engineering  (INCOSE)  Risk  Man¬ 
agement  Working  Group,  the  Project  Management  Institute  (PMI)  Risk  Management  Spe¬ 
cific  Interest  Group,  the  Camegie-Mellon  Software  Engineering  Institute  (SEI),  Software 
Program  Managers  Network  (SPM),  the  National  Aeronautic  and  Space  Agency  (NASA), 
the  Naval  Postgraduate  School  (NPS),  and  the  Defense  Systems  Management  College 
(DSMC).  The  INCOSE  Risk  Management  Working  Group  and  the  PMI  Risk  Management 
Specific  Interest  Group  [HHG+02]  collaborated  on  a  joint  project  for  a  Universal  Risk  Pro¬ 
ject  and  developed  a  list  and  definition  of  universal  risk  areas94,  applicable  to  any  type  of 
project  or  operation  in  the  private  and  government  sector95.  A  universal  risk  is  defined  as 
“an  event  or  condition  that  causes  a  deviation  from  the  plan,  with  a  reasonable  chance  of 
affecting  the  conduct  of  an  analysis  and  that  may  occur  in  any  project,  operation  or  system, 


94  ...  . 

The  three  major  risk  groups  identified  by  consensus  included:  management  risk  area,  external  risk  area,  technology  risk,  [HHG+02] 

95 

This  report  used  the  INCOSE  risk  management  process  terms  -  planning,  assessment,  handling  and  monitoring.  The  PMI  terms  for 
the  same  steps  are  -  risk  management  planning,  risk  identification,  qualitative  and  quantitative  risk  analysis,  risk  response  planning  and 
risk  monitoring  and  control  [HHG+02]. 
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regardless  of  industry,  organization  or  project/system  type”  [HHG+02],  In  addition, 
[HHG+02]  identified  and  defined  specific  risks  under  each  major  group. 

The  SEI  risk  management  process  approach  [GAB97]  consists  of  several 
functions  preformed  continuously  over  the  project  lifecycle:  identification,  analysis,  plan¬ 
ning,  tracking,  control,  and  communication.  [SPM95]  explains  an  additional  four  areas  of 
risk  classified  by  the  Software  Program  Managers  Network:  productivity,  quality,  timeli¬ 
ness,  and  user  satisfaction.  Recently,  [HDK+02]  presented  potential  improvements  to  De¬ 
partment  risk  management  of  the  operation  and  implementation  of  infonnation  systems 
supporting  vital  defense  functions  by  incorporating  an  approach  called  value  focused  think¬ 
ing.  The  NASA  independent  verification  and  validation  (IV &V)  methodology  [ConOl] 
addresses  a  risk  management  approach,  which  is  experiencing  renewed  emphasis  following 
the  less  than  successful  results  of  recent  NASA  projects  based  on  risks  introduced  into  the 
projects  by  the  implementation  of  the  “better,  faster,  cheaper”  management  process 
[OCD+02], 

Improving  reliability  of  systems  in  the  early  software  development  stages 
when  systems  are  especially  prone  to  errors  may  be  key  to  improving  quality  and  the  risk 
assessment  process  in  software  engineering.  Recent  research  at  the  NPS  by  [LNOO,  NJLOO, 
NLBOOa,  NLBOOb,  NLB+00,  NogOO]  focused  on  improving  quality  and  reducing  risk  in 
the  earliest  phases  of  the  software  development  project.  In  addition,  [NLBOOa,  NLBOOb] 
identified  the  inherent  weakness  of  human  dependencies  and  inconsistent  application  of  the 
risk  assessment  process  by  different  experts,  and  introduced  a  formal  method  with  three 
metrics  for  requirements,  personnel,  and  complexity  supporting  risk  identification. 

New  automated  risk  management  tools  continue  to  evolve.  CASE  tools 
such  as  the  Computer  Aided  Prototyping  System  (CAPS)  [LK88,  Luq89,  NLBOOa,  and 
NLBOOb]  support  a  risk  assessment-based  software  evolution  and  prototyping  approach.  A 
risk  analysis  template  [Whi99]  consists  of  elements  deemed  to  be  important  to  the  model. 
Applied  to  each  element  of  acceptability  for  the  model,  it  addresses  the  consequence  of 
missing,  poor  or  incorrect  elements.  To  manage  the  risks  introduced  by  these  factors, 
[BK02a]  introduced  the  credibility  indicator  trees  method  including  fault  tree  analysis  to 
support  defined  levels  of  V&V  and  credibility.  Risk  mitigation  techniques  directly  affect 
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the  entire  simulation  software  life  cycle,  contribute  to  improved  M&S  credibility  and  sup¬ 
port  the  accreditation  process. 

The  objective  of  the  Department’s  VV&A  process  is  to  reduce  M&S  risk 
[Kil02].  The  verification  process  reduces  the  risk  of  unintended  errors  and  the  use  of  inap¬ 
propriate  data,  while  the  validation  process  improves  confidence  that  the  M&S  outputs 
match  the  real  world,  and  that  the  level  of  data  accuracy  is  sufficient  for  the  intended  uses. 
Accreditation  reduces  the  risk  of  selecting  an  inappropriate  or  unsuitable  M&S  for  use. 
However,  well-known  V&V  process  constraints  include  the  lack  of  time,  budget  limita¬ 
tions,  an  incomplete  knowledge  of  reality,  and  complex  M&S  systems.  Moreover, 
[HHG+02]  contends  that  techniques  are  not  the  critical  issue,  but  rather  the  lack  of  man¬ 
agement  and  engineering  discipline  ensures  we  continue  to  practice  poor  paradigms  for 
system  development. 

Mission-critical  software  systems  may  present  high-risks  to  the  overall  sys¬ 
tem  development.  [CHK+90]  identifies  Mission  Critical  Computer  Resources  (MCCR)  as 
the  totality  of  computer  hardware  and  software  that  is  integral  to  a  weapon  system  along 
with  the  associated  personnel,  documentation,  supplies,  and  services.  The  DSMC  devel¬ 
oped  the  MCCR  process  for  four  reasons: 

•  Software  for  weapon  systems  is  on  the  critical  path  of  system  development, 

•  Software  can  experience  significant  development  problems  resulting  in  cost  over¬ 
runs, 

•  Modern  weapon  system  performance  depends  on  the  quality  of  the  computer  re¬ 
sources  and  the  system  is  only  as  good  as  the  software, 

•  Once  a  software  development  project  falls  behind,  adding  more  resources  (e.g., 
money,  programmers)  will  not  shorten  the  development  times  [CHK+90]. 

d.  Cost  of  Verification  and  Validation 

There  are  concerns  in  the  Department  about  the  cost  and  lack  of  implemen¬ 
tation  funding  for  conducting  credible  VV&A  programs.  Published  VV&A  costs  range 
from  five  to  eighteen  percent  of  a  projects  funds  cited  by  [GMS+96]  with  a  mode  between 
ten  and  twelve  percent  of  the  project  cost.  The  cost  of  model  validation  noted  by  [Sar98]  is 
usually  non-trivial,  particularly  when  confidence  in  the  simulation  results  is  important. 
Furthermore,  developing  absolute  validity  of  a  model  over  the  entire  domain  for  all  poten- 
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tial  uses  contends  [Sar98]  is  too  costly,  time  consuming  and  probably  impossible.  Instead, 
[Sar98]  suggests  developing  sufficient  confidence  by  executing  tests  and  evaluations  as  a 
means  for  detennining  a  model  valid  for  its  intended  application. 

The  availability  of  documentation  from  previous  VV&A  initiatives  and  un¬ 
ambiguous  acceptance  criteria  also  detennines  the  cost  of  establishing  M&S  credibility 
[Mue97b].  Addressing  the  ever-increasing  size  of  M&S,  [MBZ99]  conclude  that  the  “in¬ 
creasing  of  scale  and  degree  of  complexity  of  M&S  system[s]  add  [to]  the  difficulty  and 
cost  of  VV&A  process  in  life  cycle  of  simulation”  [MBZ99].  However,  in  a  striking  con¬ 
trast  to  [MBZ99]  cost  concept,  [Whi99]  contends  that  there  is  no  additional  cost  for  VV&A 
“since  V&A  activities  are  integral  to  normally  accepted  software  development  processes” 
[Whi99], 


e.  Return  on  Investment 

Major  investments  in  Department  M&S  have  not  credibly  established  a  re¬ 
turn  on  investment  (ROI)  cautions  [BGK+00],  and  the  challenge  to  implement  credible  and 
cost-effective  V&V  is  growing.  In  many  advanced  technologies,  providing  confidence  in 
performance  to  the  specified  level  across  the  operating  envelope  depends  to  an  unprece¬ 
dented  degree  on  confidence  in  system  M&S.  For  these  reasons,  [BGK+00]  developed  a 
business  case  framework  to  evaluate  Departmental  M&S  investment  opportunities. 

[Sta96,  BGK+00]  cites  the  failure  to  identify  a  return  on  investment  (ROI) 
for  M&S  on  the  lack  of  PM  staff  knowledge,  and  the  lack  of  Department  institutional  in¬ 
centives  to  build  expensive  models.  In  addition  [BGK+00]  compare  and  contrast  case  stud¬ 
ies  of  the  commercial  sector  Boeing  777  aircraft  and  the  Joint  Strike  Fighter  M&S  devel¬ 
opment  within  the  Department  M&S  environment.  Understanding  the  pressures  that  pro¬ 
gram  managers  are  under  to  complete  projects  on  schedule  and  within  budget,  [BGK+00] 
address  several  systemic  impediments  that  preclude  program  managers  from  expending  the 
funds,  staff,  or  time  to  investigate  the  potential  benefits  of  M&S.  In  order  to  improve  this 
situation  [BGK+00]  develop  a  seven-step  procedure  to  build  a  M&S  business  case,  and  re¬ 
iterate  a  recurring  theme  found  in  the  body  of  research;  recommending  a  disciplined  ap¬ 
proach  and  methodology. 
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F. 


SUMMARY 


Chapter  III  introduced  M&S  credibility  issues  and  identified  concerns  that  decision¬ 
makers  have  with  Department  simulation  results,  followed  by  an  overview  of  the  Depart¬ 
ments  VV&A  processes,  and  several  factors  contributing  to  the  implementation  of,  or 
shortfalls  of  the  VV&A  processes  in  the  Department.  Two  major  periods,  the  Growth  and 
the  Managed  Eras  characterize  the  initiatives,  factors,  and  efforts  to  improve  credibility  in 
the  Department’s  simulation  results. 

This  chapter  also  addressed  the  credibility  of  M&S  in  the  context  of  the  procedures, 
policies  and  programs  the  Department  established  to  improve  decision-makers’  level  of 
confidence  in  the  results  produced  by  the  M&S.  It  is  clear  from  the  research  that  Depart¬ 
ment  and  the  Service  Components  acted  with  due  diligence  to  implement  the  policy,  proce¬ 
dures,  and  guidelines  to  improve  the  quality  of  Department  M&S.  By  the  mid-1990s  the 
Department  established  M&S  management  organizations,  developed  policies,  implemented 
plans,  allocated  significant  funding,  and  executed  major  new  programs  with  the  goals  and 
objectives  of  improving  Department  M&S  practices.  However,  not  withstanding  these 
considerable  efforts,  major  concerns  still  exist  about  the  credibility  of  the  Department 
simulation  process.  There  are  several  major  reasons  for  this  lack  of  credibility  in  Depart¬ 
ment  M&S  identified  in  the  chapter. 

The  new  Department  M&S  management  organizations,  such  as  the  DMSO,  had  to 
overcome  Service  parochialism,  the  unpredictability  of  the  Department  budget  process,  the 
vagaries  of  the  Department  acquisition  process,  and  the  life  cycle  management  of  over 
forty  years  of  unplanned  Department  M&S  growth.  Then  in  the  late  1990s  significant 
funding  and  manpower  resources  originally  programmed  to  improve  Department  M&S 
were  committed  to  the  Year  2000  problem  resolution  project.  Research  indicates  three  ma¬ 
jor  categories  of  technical,  cultural,  and  managerial  challenges  collectively  hindered  the 
development  of  improved  Department  M&S  credibility. 

The  research  also  identified  an  expanding  role  for  M&S  at  the  national  policy  and 
strategic  level,  and  within  the  Department  five  major  domains.  A  synthesized  M&S  Credi¬ 
bility  Taxonomy  established  at  Figure  3-1  identified  interrelated  factors  affecting  the  De- 
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partment’s  objectives  for  improved  credibility,  especially  in  the  area  of  authoritative  repre¬ 
sentations.  There  are  many  other  factors  involved  in  simulation  credibility;  however,  these 
factors  directly  support  the  development  of  software-intensive  simulations. 

The  Department  established  the  VV&A  process  in  the  mid-1990s  as  the  primary 
method  for  establishing  M&S  credibility,  prescribed  regulatory  requirements  and  provided 
best  practices  and  techniques  to  the  community.  The  Department  continually  cited  a  lack 
of  credibility  in  M&S  stemming  from  inadequate  VV&A  implementation  negating  deci¬ 
sion-makers’  confidence  levels  in  the  simulation  results  as  a  systemic  issue  needing  major 
improvements,  especially  in  the  Department’s  developmental  and  operational  test  commu¬ 
nities.  Companion  assessments  of  VV&A  process  implementation  in  the  private  sector  and 
other  public  sectors  indicated  that  the  VV&A  process  failed  to  deliver  the  desired  results. 

We  identified  and  discussed  VV&A  principles  illustrated  in  Figure  3-2,  including 
the  critical  role  that  data  plays  in  the  VV&A  process.  However,  the  lack  of  authoritative 
data  is  a  major  deficiency  in  the  Department’s  attempt  to  improve  M&S  credibility.  Activi¬ 
ties  supporting  credence  in  the  M&S  included  model  verification,  conceptual  model  valida¬ 
tion,  operational  validity,  data  validity  (Figure  3-3)  and  a  model  range  of  accuracy.  Five 
key  factors  supported  model  credibility:  capability,  software  accuracy,  data  accuracy,  re¬ 
sults  accuracy,  and  usability.  The  most  definitive  factor  supporting  credibility  is  to  com¬ 
pare  the  model  output  with  the  actual  or  proposed  system  output  data.  We  also  identified 
and  summarized  common  verification  and  validation  techniques  in  Figure  3-4  employed  by 
the  Department,  including  M&S  V&V  techniques  (e.g.,  informal,  static,  dynamic,  statisti¬ 
cal,  fonnal)  identified  in  the  [GMS+96  and  RPGOO].  We  noted  that  systemic  issues  identi¬ 
fied  in  by  reports,  studies,  and  assessments  the  early  1980s  are  still  evident. 

The  Department’s  foundation  for  M&S  credibility  supporting  confidence  in  simula¬ 
tion  results  is  the  M&S  verification  and  validation  process.  Verified  and  validated  concep¬ 
tual  model  support  credible  V&V  processes.  However,  many  consider  V  &  V  too  expen¬ 
sive  and  time  consuming.  In  this  research  we  reviewed  the  V&V  process  in  the  context  of 
supporting  Department-  and  National-level  decisions  with  improved  credibility  and  confi¬ 
dence,  and  confirmed  previous  assessments  indicating  the  quality  of  Department  V&V 
practices  are  ad  hoc  and  inconsistently  applied. 
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The  conceptual  model  of  the  mission  space  (CMMS  or  FDMS)  is  a  critical  compo¬ 
nent  of  a  rigorous  Department  verification  and  validation  program.  The  M&S  community 
developed  many  conceptual  model  formats  [RPGOO],  however,  there  is  no  consensus  on  a 
standard  conceptual  model  at  this  time.  In  this  research  we  identified  the  underlying  issues 
and  propose  a  model  supported  by  the  software  engineering  process  and  a  better  under¬ 
standing  of  the  complexity  of  the  real-world  fidelity  definition  and  measurement.  This  re¬ 
search  also  expanded  on  the  work  accomplished  by  the  SISO  Fidelity  Experimentation  Im¬ 
plementation  Study  Group  and  their  work  on  the  Fidelity  Conceptual  Framework. 
[RGHOO], 

During  the  research,  several  common  themes  continually  emerged  from  the  litera¬ 
ture:  reuse,  the  lack  of  documentation,  risk  management,  cost,  and  the  identification  of  a 
quantifiable  return  on  investment  (ROI)  from  M&S.  Reuse  initiatives  provided  mostly  dis¬ 
appointing  results;  however,  “large-scale”  reuse  initiatives  have  the  potential  to  improve 
this  process.  Documentation  is  a  consistent  problem  in  many  software  development  pro¬ 
jects.  In  the  M&S  area  in  addition  to  the  lack  of  software  documentation,  the  lack  of  con¬ 
ceptual  model  documentation,  coupled  with  the  lack  of  documented  V&V  activities  se¬ 
verely  undennined  the  Department’s  efforts  to  improve  credibility.  Risk  management  is  a 
continuous  activity  applied  during  the  entire  life  cycle  of  the  M&S,  where  the  early  identi¬ 
fication  and  resolution  of  risks  may  be  critical  to  improving  overall  M&S  quality  and 
credibility,  appears  uneven. 

The  cost  to  accomplish  V&V,  constantly  cited  as  being  too  high,  may  continue  to 
grow  as  the  M&S  become  larger  and  more  complex.  In  addition,  dentifying  a  quantifiable 
ROI  has  been  an  elusive  goal  for  the  Department,  and  as  a  result,  Department  acquisition 
program  managers  may  be  reluctant  to  invest  the  required  resources  in  a  tool,  which  re¬ 
quires  significant  resources  today  for  questionable  results  tomorrow. 

The  Department’s  legacy  M&S  portfolio  is  a  major  study  factor.  The  Department 
developed  a  significant  and  expensive  portfolio  of  legacy  M&S  over  the  past  forty  years. 
Although  an  operational  system  has  a  de  facto  architecture,  many  of  these  M&S  evolved  in 
an  ad  hoc  manner  and  have  become  large  legacy  systems  with  multiple  stakeholders  and 
expensive  support  infrastructures.  A  significant  number  of  these  systems  lack  conceptual 
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models  and  have  an  ad  hoc  V&V  history.  In  some  cases  the  developer  created  conceptual 
models  after  the  fact,  if  at  all,  and  adherence  to  disciplined  M&S  V&V  process  was  a  low 
priority.  Department  agencies  may  also  have  a  tendency  to  use  these  systems  for  studies  or 
tests  for  which  they  are  ill  suited,  in  order  to  show  a  return  on  investment.  As  a  result  the 
acceptability  criteria  and  caveats  developed  by  the  accreditation  process  for  these  simula¬ 
tions  may  be  of  limited  value  or  counter-productive  to  the  original  purpose  and  intent  of  the 
study  or  test  objectives,  and  fail  to  establish  credibility  in  the  simulations  or  confidence  in 
the  simulation  results 
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IV.  HETEROGENEOUS  SYSTEM  REPRESENTATION  ANOMALIES 


A.  INTRODUCTION 

Improving  the  credibility  of  large-scale,  legacy  Department  M&S  requires  an  un¬ 
derstanding  of  the  underlying  systemic  causes  generating  the  pre-conditions  for  heteroge¬ 
neous  system  representation  anomalies,  especially  in  federation  interoperability.  Chapter 
IV  reviews  several  heterogeneous  challenges  to  Department  M&S  credibility.  These  di¬ 
verse  challenges  include  syntactic  and  semantic  heterogeneity,  data  heterogeneity,  com¬ 
plexity,  interoperability,  technical  and  substantive  interoperability,  fidelity  heterogeneity, 
and  multiresolution  modeling  heterogeneity. 

B.  ELUSIVE  AUTHORITATIVE  REPRESENTATIONS 

Authoritative  representations  have  been  consistently  identified  as  major  objectives 
in  Department  M&S,  yet  few  of  the  [DoD95,  DoDOla]  objectives  for  authoritative  repre¬ 
sentations  have  been  achieved  [MosOO,  CraOla,  CraOlb].  As  a  result  of  the  long-standing 
unresolved  systemic  issues  associated  with  Department  M&S  senior  civilian  and  military 
decision-makers  have  significant  reservations  that  Department  M&S,  as  currently  prac¬ 
ticed,  can  meet  the  serious  future  demands  generated  by  national  security  requirements  for 
M&S  in  the  Department.  In  1987,  [Che87]  placed  boundaries  on  simulation  results,  later 
confirmed  by  [San97a]  noting: 

Although  simulations  are  useful  tools,  they  are  always  approximations  to  re¬ 
ality,  and,  therefore,  the  credibility-the  level  of  confidence  that  a  decision¬ 
maker  should  have  in  their  results— is  open  to  question  [Che87]. 

Current  Department  M&S  users  still  require  accurate,  interoperable  elements  of  the 
mission-space  and  the  draft  [DoDOla]  retains  authoritative  representations  as  a  Depart¬ 
ment-wide  M&S  objective.  In  addition  [DoDOla]  lists  expanded  requirements  for  credible, 
authoritative  representations  including  U.S.,  allied,  friendly,  coalition,  neutral  and  threat 
and  paramilitary  system  representations  for  C4ISR,  combat,  combat  support  and  combat 
service  support  systems  and  processes,  supported  by  standard  taxonomies  and  common  ob¬ 
ject  classes  for  systems  representations  by  FY2004.  Successful  attainment  of  these  inter¬ 
operability  objectives  by  the  Department  and  international  M&S  communities  assumes  the 
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acceptable  resolution  of  many  heterogeneity96  issues,  including  abstractions,  networks, 
federations  and  representations. 

Abstraction  is  an  intrinsic  human  activity  employed  to  reduce  or  filter  the  full 
complexity  of  the  real  world  to  manageable  proportions  by  deleting  unnecessary  detail  in 
order  to  provide  data  in  context  (e.g.,  infonnation)  supporting  the  decision-making  process 
[HJS97].  Abstraction  mechanisms  [Til98]  selectively  emphasize  the  level  of  detail,  em¬ 
phasizing  the  details  pertinent  to  the  intended  use,  while  hiding  unneeded  detail.  [Sow88, 
Til98]  discussed  common  abstraction  mechanisms  including:  classification,  aggregation, 
and  generalization: 

•  Classification  is  a  form  of  abstraction  establishing  an  instance-of  relationship  be¬ 
tween  the  object  type  in  the  schema  and  its  instance  with  an  object  defined  as  a  set 
of  instances  with  shared  common  characteristics, 

•  Aggregation  is  a  form  of  abstraction  establishing  a  part-of  relationship  between  the 
component  objects  and  the  aggregate  object,  with  relationships  between  objects 
considered  at  a  higher  level  aggregate  object  and  specific  details  of  the  constituent 
objects  hidden, 

•  Generalization  is  a  form  of  abstraction  establishing  an  is-a  relationship  between  the 
specialized  and  generic  object,  when  similar  objects  relate  to  a  higher-level  generic 
object  and  the  constituent  objects  considered  specializations  of  the  generic  object 
[Sow88,  Til98], 

C.  DATA  MODEL  HETEROGENITY  AND  INTEROPERABILITY 


Data  models  establish  the  essential  properties  and  well-defined  relationships  be¬ 
tween  raw  data  in  a  system  in  a  form,  which  supports  efficient  storage,  timely  retrieval  of 
data,  and  provides  the  basis  for  tools  and  techniques  to  support  data  modeling  [RagOl].  A 
data  model  captures  the  static  and  dynamic  characteristics  supporting  data-related  proc¬ 
esses,  with  static  properties  defined  in  a  database  schema  and  the  dynamic  characteristics 
developed  into  specifications  for  reports,  queries,  and  transactions.  The  schema  maintains 
the  object  type  definitions,  attributes,  relationships  and  static  constraints  of  the  data  reposi¬ 
tory  or  database,  an  instance  of  the  schema. 
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Heterogeneous  is  defined  as:  consisting  of  or  involving  dissimilar  elements  or  parts.  Heterogeneous  networks  are  a  collection  of  simu¬ 
lations  with  partially  consistent  behaviors  and/or  partially  correlated  databases.  Examples  include  simulators  of  different  fidelity,  mixed 

virtual  and  live  simulations,  and  mixes  of  virtual  and  constructive  simulations  [DoD98]. 

97 

Static  characteristics  include  objects,  attributes,  and  relationships  among  objects  [Til98]. 

98 

Dynamic  characteristics  include  operations  on  objects,  operation  properties,  and  relationships  among  operations  [Til98]. 
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The  most  significant  record-based  logical  data  models  includes  the  hierarchical  data 
model,  organized  into  simple  tree  structures;  the  network  data  model,  a  superset  of  the  hi¬ 
erarchical  data  model,  without  the  requirement  for  tree  structures;  and  the  relational  data 
model,  based  on  a  mathematical  foundation  of  a  relation  (e.g.,  a  set  of  n-tuples)  organized 
as  a  related  collection  of  tables  [01183].  These  classic  highly  machine-oriented,  and  record- 
oriented  data  models,  or  syntactic  data  models  [Til98],  are  well  suited  to  the  computer  en¬ 
vironment  and  organized  for  optimal  efficiency  supporting  storage  and  retrieval  operations, 
however  these  models  lack  semantic  relativism"  and  abstraction  mechanisms100  required 
for  dealing  with  complexity  in  large-scale  systems  [Til98] . 

[Til98]  describes  a  second  category,  semantic  data  models,  which  combine  basic 
knowledge  representation  techniques  with  database  technology,  and  represent  a  transition 
from  the  basic  record-oriented  approach  towards  a  model  supporting  more  human-oriented 
semantic  constructs.  While  traditional  data  models  store  data,  and  semantic  models  shift 
towards  a  more  human  knowledge  model,  [Til98]  suggests  a  third  method,  the  conceptual 
modeling 101  activity  geared  more  towards  the  domain-level  knowledge  and  program  under¬ 
standing,  with  a  focus  on  the  end-user.  Program  understanding  supports  development  of  a 
fidelity-based  product  line  conceptual  model  referent  domain-architecture,  since  large  leg¬ 
acy  systems  represent  substantial  corporate  knowledge  of  an  organization’s  business  rules, 
requirements,  and  design  decisions.  [Til98]  further  suggests  that  all  three  techniques  may 
be  more  suitable  than  one  method  if  there  are  different  categories  of  artifacts  (e.g.,  data, 
knowledge,  infonnation),  requiring  a  relational  model  physical  storage  of  data,  a  concep¬ 
tual  model  for  representing  domain-level  knowledge,  or  a  semantic  model  for  interactive 
discovery. 


Semantic  relativism  is  the  ability  to  view  the  elements  and  concepts  representing  a  modeled  system  from  different  perspectives  based 
on  the  application  [Til98]. 

The  most  common  abstraction  mechanisms  are  classification,  a  form  of  abstraction  in  which  an  object  is  defined  as  a  set  of  instances; 
aggregation;  and  generalization,  a  form  of  abstraction  in  which  similar  objects  are  related  to  a  higher  level  generic  object  [Sow88] 
Conceptual  modeling  (e.g.,  conceptual  schemata)  in  this  context  is  the  activity  of  formally  describing  aspects  of  some  information 
space  for  the  purpose  of  understanding  and  communicating,  with  an  emphasis  on  the  knowledge  organization  used  by  humans,  rather 
than  the  data  organization  used  by  machines  [Til98] . 
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D.  HETEROGENEOUS  SYNTACTIC  DATA  MODELS 


[Wie99]  submits  the  objective  of  interoperation  is  to  increase  the  value  of  informa¬ 
tion  when  accessing,  relating,  and  combining  infonnation  from  multiple  sources.  In  addi¬ 
tion,  [Wie90,  Wie93,  Wie96a]  acknowledged  that  data  comes  from  many  diverse  and  het¬ 
erogeneous  sources  and  further  suggested  that  joining  heterogeneous  data  is  necessary  to 
generate  information  or  compose  large-scale  software  applications  [MBS+99]. 

[Wie90,  Wie93,  Wie96a]  also  addressed  several  contributing  factors  for  differences 
among  systems  in  their  research  on  database  schema  integration  including  the  different  per¬ 
spectives  of  the  developers  and  users;  the  use  of  equivalent  constructs  (e.g.,  different  uses 
of  the  same  constructs  to  create  the  same  representation);  and  incompatible  design  specifi¬ 
cations.  Focusing  primarily  on  database  constructs  and  schema  research,  [WW90,  Wie93, 
Wie96a,  WieOOa,  WieOOb]  developed  eight  viewpoints  for  differences  among  systems,  and 
[You02c]  added  a  ninth  viewpoint  of  heterogeneity  gennane  to  this  research: 

•  Heterogeneity  of  Hardware  and  Software  systems, 

•  Heterogeneity  of  Organizational  Models  including  network,  hierarchical,  relational, 
universal  database  models  and  object-structured  data, 

•  Heterogeneity  in  Representation  since  it  causes  anomalies  at  the  limits  or  bounda¬ 
ries  (e.g.  5-digit  zip  code  versus  9-digit  zip  code), 

•  Heterogeneity  of  Scope  with  different  databases  capturing  different  views  of  the 
same  real-world  object  (e.g.,  employees  paid  versus  employees  available), 

•  Heterogeneity  of  Level  of  Abstraction  manifested  by  different  aggregation  schemes 
(e.g.,  personal  income  versus  family  income),  implying  the  existence  of  an  aggrega¬ 
tion  hierarchy, 

•  Heterogeneity  of  Meaning  addressing  assignment  of  attribute  values  to  real-world 
attributes  (e.g.,  postal  codes  versus  town  names), 

•  Heterogeneity  of  Temporal  Validity  caused  when  values  have,  often  implicitly,  dif¬ 
ferent  temporal  ranges  (e.g.,  monthly  budget  versus  weekly  production),  which  may 
not  support  aggregation  [Wie93], 

•  Semantic  Heterogeneity  occurs  because  the  meaning  of  words  depends  on  the  con¬ 
text,  and  data  sources  develop  within  their  own  context.  Types  of  Semantic  Het¬ 
erogeneity  include  synonyms,  spelling  differences,  and  the  use  of  identically  spelled 
words  to  refer  to  different  objects  [WieOOa,  WieOOb], 

•  Heterogeneity  of  Structure  causing  a  variation  in  the  structure  of  the  information  ar¬ 
rangement  between  systems  employing  the  same  organizational  model  (e.g.,  a  plane 
modeled  as  an  attribute  in  one  system  and  an  object  in  another  [You02c]. 
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In  supporting  a  decision-maker,  [Wie02]  suggests  access  to  a  variety  of  methods 
and  heterogeneous  information  sources  for  predicting  the  future  including  extrapolation, 
interpolation,  projections,  published  projections,  spreadsheets,  discrete  simulations,  and 
continuously  executing  simulations  (e.g.,  weather  predictions).  Many  methods  are  compu¬ 
tation  intensive,  and  except  for  desktop  models  such  as  spreadsheets,  most  simulations  are 
too  complex,  costly,  or  require  too  much  manpower  or  time  to  meet  decision-makers  re¬ 
quirements  for  infonnation  supporting  analysis  of  alternate  future  courses  of  action.  In  ad¬ 
dition,  distributed  simulations  employing  HLA,  DIS,  or  ALSP  protocols  rarely  interface 
with  other  external  types  of  infonnation  systems  such  as  databases,  although  simulation 
systems  may  include  internal  databases  or  files  to  retain  past,  and  near-current  data  for  ta¬ 
ble  lookups  and  scenario  support  to  the  simulation.  Relational  databases  support  the  Struc¬ 
tured  Query  Language  (SQL)  for  database  access;  however,  SQL  currently  has  no  native 
capability  to  communicate  with  a  simulation  or  a  federation. 

[WieOOa,  WieOOb]  notes  that  semantic  heterogeneity  causes  failure  to  find  the  de¬ 
sired  object,  and  affects  the  level  of  precision  in  selection,  aggregation,  and  comparison 
when  integrating  infonnation.  Furthermore,  [WieOOa,  WieOOb]  believes  a  global  solution 
to  semantic  mismatch  is  unfeasible  due  to  the  number  of  participants,  terminology  changes, 
and  scope,  but  notes  that  precise,  finely  differentiated  terms  and  abbreviations  may  be  suit¬ 
able  for  domain  efficiency.  For  instance,  [WieOOa,  WieOOb]  opines  that  consistency  is  lo¬ 
cal  (e.g.,  precision  for  an  on-line  commerce  requires  a  consistent  structure  and  a  consistent 
terminology  for  the  same  set  of  objects). 

[WW90,  Wie93,  and  MW02]  suggest  that  knowledge  of  the  enterprise  operation 
supports  the  development  of  the  best  possible  information  from  diverse  sources,  and  also 
note  that  since  heterogeneity  is  never  completely  resolved,  uncertainty  heightens  in  the  de¬ 
cision-making  process.  In  the  same  vein,  [WieOOa]  suggests  that  many  small-scale  efforts 
provide  superior  results  over  major  standardization  efforts,  and  proposes  that  precision  and 
relevance  attributes  will  be  more  valuable  than  completeness  and  recall  attributes.  Preci¬ 
sion,  with  its  many  components  (e.g.,  rule-based  matching,  metadata,  quality  attributes,  and 
processing  models)  is  an  important  aspect  of  information,  and  will  become  increasingly 
important  [WieOOa], 
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[Wie99]  also  notes  that  interface  standards  are  critical  for  building  and  maintaining 
multi-layer  systems,  citing  the  role  of  SQL  for  structured  tables,  CORBA  and  DCOM  for 
middleware;  and  XML  when  data  cannot  be  structured  well.  [Wie99  and  DM03]  further 
suggest  using  XML  for  specific  domains  with  domain-specific  type  descriptions  or  sche¬ 
mas,  to  help  in  matching  the  meaning  of  shared  infonnation. 

As  a  possible  solution  to  the  heterogeneity  challenge,  [WW90,  Wie97a,  Wie99, 
WieOOa,  WieOOb]  proposes  a  two-dimensioned  mediator  architecture  partitioning  resources 
and  services  to  resolve  heterogeneity  issues  with  three  horizontal  layers  (e.g.,  client  appli¬ 
cations,  intermediate  service  modules,  and  base  servers),  and  vertically  into  any  domains, 
with  a  limited  number  of  supporting  servers.  Noting  that  any  implementation  for  a  system 
presenting  past  and  future  infonnation  requires  the  cooperation  of  specialists  in  distinct 
fields  and  an  openness  to  learn  new  methods  and  techniques,  [Wie96a,  Wie97a,  Wie02] 
identified  a  three  layered  hierarchy,  with  an  intelligent  middleware  mediator  layer  linking 
the  heterogeneous  information  sources  at  the  bottom  with  the  application  programs  at  the 
top  to  achieve  an  intelligent  integration  of  information.  The  mediator  performs  the  follow¬ 
ing  tasks: 

•  Accessing  and  retrieving  relevant  data  from  multiple  heterogeneous  information 
sources, 

•  Abstracting  and  transforming  retrieved  data  into  a  common  representation  and  se¬ 
mantics, 

•  Integrating  the  homogenized  data  according  to  matching  keys, 

•  Reducing  the  integrated  data  by  abstraction  to  increase  the  information  density  in 
the  transmitted  result  [Wie97a], 

[Wie97a,  WJ98,  Wie99,  Wie02]  demonstrated  a  capability  for  supporting  integrated 
predictive  capabilities  from  spreadsheets  and  other  simulation  tools  into  infonnation  sys¬ 
tems,  employing  an  interface  language,  SimQL,  combined  with  wrappers.  SimQL  provides 
a  schema,  describing  the  accessible  contents,  and  a  query  language  to  access  the  informa¬ 
tion  resources,  similar  to  SQL  capabilities  [Wie99]  and  has  four  major  components: 

•  A  compiler  for  the  SimQL  language,  generating  code  to  access  wrapped  resources, 

•  A  repository  containing  the  schemas  for  the  wrapped  resources  with  input  and  out¬ 
put  parameters  identified, 

•  A  wrapper  generation  tool  to  bring  simulations  [WJ98],  spreadsheets,  and  web  re¬ 
sources  into  compliance, 
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•  The  source  forecasting  tools,  spreadsheets,  discrete  simulations,  and  web  sources 

[Wie99], 

Other  methods  for  composing  components  from  heterogeneous,  autonomous,  and  distrib¬ 
uted  information  sources  include  CP  AM  [MBS+99],  an  image  indexing  and  retrieval  algo¬ 
rithm,  WBIIS,  providing  a  search  capability  for  a  large  image  database  [WWF+98],  and  the 
Object-Oriented  Method  for  Interoperability  (OOMI)  [You02c]. 

The  [You02c]  solution  is  a  federated  interoperability  object  model  (FIOM)  and  an 
automatic  translator  generator.  [YBG+01,  You02c]  developed  the  FIOM,  an  interoperabil¬ 
ity  object  model  defined  for  a  specific  federation  of  systems,  and  an  interoperability  object 
model  for  interoperability  (OOMI)  with  a  corresponding  structure.  [YBG+01,  You02c] 
also  envisions  the  OOMI  as  an  extension  of  the  contemporary  object  model  with  a  class 
structure  property  and  extended  attribute  and  operation  properties  to  capture  the  different 
representations.  The  [YBG+01,  You02c]  translator  function  is  a  software  wrapper,  com¬ 
ponent  of  a  message-based  architecture,  or  as  part  of  the  data  store,  with  an  XML-based 
message  translation  currently  under  study  for  implementing  the  proposed  model.  Where 
[You02c]  provides  a  necessary  near-term,  interim  solution  for  interoperability  via  the  Ob¬ 
ject-Oriented  Method  for  Interoperability  (OOMI)  translators,  this  research  addresses  the 
root  cause  issues  that  effect  simulation  substantive  interoperability  in  high-risk,  mission 
critical  simulation  systems,  and  addresses  the  issue  at  the  first  level  of  abstraction,  the  ar¬ 
chitecture. 

Research  continues  reveal  mew  methods  to  fuse  infonnation  from  heterogeneous 
resources  including  databases,  text,  semi-structured  infonnation  bases  [WW90,  Wie97a] 
involving  intrinsic  semantic  differences  [Wie03],  supported  by  complementing  ontology102 
research  [Gua98,  MW02],  Commercial  software  vendors  also  continue  to  expand  basic 
database  systems,  adding  communication,  data  mining,  and  analysis  functionality  to  the 
basic  database  capabilities  supporting  the  decision-makers  need  to  plan  and  schedule  future 
actions,  and  assess  the  effects  of  alternate  future  courses  of  action  [Wie99].  Data  reposi¬ 
tory  holdings  of  past  and  near-current  data  provide  only  limited  background  information 

102  The  semantics  of  an  information  source  is  captured  by  its  ontology,  the  collection  of  domain  terms  and 
their  relationships  used  in  the  discourse  [MW02]. 
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for  assessing  alternate  future  courses  of  action,  while  decision-makers  require  a  spectrum 
of  heterogeneous  information  including  past,  present,  and  likely  future  situations  [Wie99, 
DoDOla,  Wie02,  Wie03].  Simulations,  coupled  with  other  heterogeneous  information  re¬ 
sources,  may  provide  a  potential  new  information  source  to  support  a  decision-makers 
analysis  of  alternate  future  courses  of  action. 

E.  HETEROGENEOUS  CONCEPTUAL  MODELS 

The  Department  addressed  many  M&S  problems  in  the  1990’s  to  improve  overall 
quality,  including  technical  and  substantive  interoperability.  Although  there  has  been  pro¬ 
gress  with  the  technical  interoperability  problem  since  the  mid-1990s,  many  approaches 
failed  to  improve  the  credibility  of  authoritative  representations  and  reduce  the  federation’s 
interoperability  problems  caused  by  substantive  interoperability  anomalies.  This  results 
from  simulation  developers  abstracting  real-world  objects  in  their  model  representations 
from  different  perspectives,  requirements,  constraints,  and  objectives. 

Over  time  developers  modeled  many  of  the  same  objects  in  different  ways,  poten¬ 
tially  causing  problems  when  they  interact  in  a  federation.  [DSB+99,  HYOla,  HYOlb, 
YPH01]  identified  systemic  substantive  interoperability  issues  with  legacy  simulation 
software  systems  and  the  inconsistent  representation  of  the  same  entity  in  multiple  systems. 
There  is  a  very  close  correlation  on  the  root  causes  between  the  technical  and  substantive 
interoperability  issues  cited  in  the  simulation  domain,  and  the  previously  cited  work  of 
[Wie93,  KinOl,  You02c]  on  developing  infonnation  from  heterogeneous  data  sources. 

This  research  revealed  significant  progress  in  the  area  of  technical  interoperability, 
including  research  into  multilevel  simulation  of  discrete  network  models  [Ham96],  distrib¬ 
uted  simulations  [Ham96,  HNP97],  and  the  Department-sponsored  development  of  the 
HLA.  These  advances  supported  the  ability  of  federates  to  physically  connect  and  ex¬ 
change  data  employing  a  federation  object  model,  the  use  of  common  standards,  and  coor¬ 
dinated  data  structures.  The  HLA  infrastructure  supports  hardware  and  software  compati¬ 
bility,  standards  compatibility,  time  management,  security,  and  coordinated  use  of  the  RTI. 

However,  federation  interoperation  currently  requires  significant  manpower  and  co¬ 
ordination,  and  often  encounters  technical  and  substantive  interoperability  issues  during 
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operation.  Substantive  interoperability  has  been  a  systemic  issue  since  the  Department’s 
earliest  experiences  with  networking  simulations,  SIMNET,  and  persisted  through  the  evo¬ 
lution  of  the  ALSP,  DIS  and  HLA  protocols.  Today,  substantive  interoperability,  including 
entity  interoperability  issues  remains  a  major  challenge  for  achieving  truly  distributed 
simulation  interoperability  [DoD03a].  Entity  interoperability  includes  a  logical  interaction 
between  entities  modeled  in  different  simulations,  temporal  resolution,  spatial  resolution, 
and  a  coherent  relationship  between  the  components  of  the  physical  environment  (e.g., 
ground,  ocean,  atmosphere,  weather,  and  the  electro-magnetic  spectrum)  to  support  the  in¬ 
teroperability  context  [YPH01].  Entity  interoperability  also  encompasses  the  level  of  rep- 
resentation  ,  attributes  and  behavior  including: 

•  The  required  federation  entities  are  available  at  the  necessary  level  of  detail, 

•  The  federation  entities  fit  together, 

•  The  required  federation  entities  meet  the  critical  characteristics  of  the  real-world  en¬ 
tities, 

•  The  required  federation  entities  possess  the  salient  attributes  supporting  the  federa¬ 
tion’s  intended  use, 

•  The  required  federation  entities  accurately  model  the  needed  behavior, 

•  The  required  federation  entities  work  together  logically, 

•  The  required  federation  entities  employ  consistent  algorithms  to  compute  effects 
[YPH01,  HYOla]. 

The  RTI  time  management  services  support  federation  synchronization  of  individ¬ 
ual  federate  time  and  attempts  to  manage  federation  temporal  resolution  assuming: 

•  Each  federate  computes  change  in  the  entity  states  at  some  resolution  of  simulation 
time, 

•  Incremental  state  changes  occur  often  enough  to  support  logical  interaction  between 
entities,  and  allow  the  exchange  of  data  to  occur  [YPH01,  HYOla], 

The  spatial  resolution  shared  among  the  federates  must  also  support  the  intended 
use,  although  the  federates  are  not  required  to  compute  the  or  store  the  entity  spatial  data  in 
the  same  reference  frame  assuming: 

•  Each  federate  computes  the  location  of  entities  on  a  specific  geospatial  reference 
system, 


Level  of  detail  -  Entities  do  not  require  the  same  resolution,  as  long  as  the  interactions  between  entities  meet  the  intended  use  of  the 
federation  [YPH01]. 
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•  The  computed  and  shared  entity  locations  allow  different  federates  to  adequately 
and  consistently  meet  required  federation  geospatial  and  other  environment  objec¬ 
tives  [YPHOla]. 

The  environment  is  a  critical  factor  for  achieving  substantive  interoperability,  with 
each  federate  developing  a  sufficiently  consistent  correlated  view  of  the  environment,  in¬ 
cluding  both  visual  (e.g.,  tanks,  planes,  buildings,  water)  and  non-visual  spectrums  (e.g., 
radio  frequency  (RF),  infrared  radiation  (IR),  photons  (lasers)).  Early  efforts  enforced  ho¬ 
mogeneity  by  developing  a  mapping  between  the  different  simulation  environmental  repre¬ 
sentations,  while  today  there  are  integrated  data  set  solutions  including  a  standard  syntax 
and  semantics  with  the  SEDRIS  data  reference  model  providing  a  standard  interface  speci¬ 
fication,  data  dictionary,  and  data  coding  standards  [YPH01].  Research  continues  in  repre¬ 
senting  the  effects  of  the  environment  on  systems  and  human  behavior  (e.g.,  the  detection 
capability  of  an  IR  or  RF  sensor  in  a  perturbed  environment),  and  conversely  on  the  impact 
of  systems  and  humans  on  the  environment  (e.g.,  the  impact  of  chemical  or  radiation  weap¬ 
ons  in  a  city). 

A  few  analogous  examples  of  the  current  challenges  presented  by  the  lack  of  tech¬ 
nical  and  substantive  interoperability  provided  below  illustrate,  compare,  and  contrast  the 
concepts  of  technical  and  substantive  interoperability: 

•  You  dial  a  number  for  an  important  business  call,  but  accidentally  reach  a  computer 
modem  and  hear  the  sound  of  a  modem  trying  to  synchronize  with  a  sending  mo¬ 
dem.  The  communication  network  perfonned  as  designed.  The  communication 
protocols  worked  up  to  a  point,  however,  you  received  no  message.  This  represents 
a  passing  grade  for  technical  interoperability,  but  a  lack  of  substantive  interopera¬ 
bility.  The  result:  time  lost  redialing,  maybe  lost  business  since  the  message  did  not 
get  through  on  time. 

•  In  another  example,  arriving  home  after  work  and  just  as  you  prepare  to  relax  for  a 
late  dinner,  the  phone  rings  and  you  are  greeted  by  a  telemarketer  or  worse  yet,  no 
response,  indicating  an  autodial  technique  advancing  ahead  of  the  telemarketer’s 
verbal  proposal.  In  this  case  you  hung  up  the  receiver  even  though  the  communica¬ 
tion  network  and  protocols  worked  exactly  as  designed.  This  case  illustrates  suc¬ 
cessful  technical  interoperability  and  a  lack  of  substantive  interoperability,  how¬ 
ever,  since  the  telemarketer1  s  message  failed  to  go  through.  The  result:  no  message 
delivered  and  no  sale  for  the  telemarketer. 

•  In  an  HLA  federation  scenario  the  publisher  submits  a  number  0.506127  for  a  state 
variable  in  a  specific  instance,  and  the  federation  subscription  process  apparently 
works.  Initial  analysis  suggests  the  results  are  normal.  Subsequently  following  a 

88 


major  live  test  event  such  as  a  flight  test,  post-test  simulations  runs  execute,  but  er¬ 
rors  in  process  truncate  the  value  to  three  digits,  e.g.,  0.506  from  the  0.506127 
number  needed  for  the  required  precision.  Multiple  runs  of  the  simulation  execute, 
but  the  results  appear  inconsistent.  Several  factors  in  the  host  processing  systems 
including  translation,  level  of  precision,  or  even  perturbations  due  to  chaos  theory104 
may  be  the  root  cause.  The  result:  substantive  interoperability  anomalies  in  an 
HLA  federation,  inconsistent  results,  lack  of  credibility  in  the  simulation  and  loss  of 
confidence  by  the  user. 

Substantive  interoperability  anomalies  occur  when  one  of  the  following  types  of 
representational  anomalies  manifests  themselves:  1)  event  phase  anomalies,  2)  event  order¬ 
ing  anomalies,  3)  state  error  anomalies  or  registration  anomalies105  [HYOla,  YPH01]  and 
create  intolerable  representational  anomalies.  Representational  anomalies  beyond  accept¬ 
able  tolerances  for  the  federation’s  intended  purpose  can  manifest  themselves  from  an  inva¬ 
lid  federate  and/or  from  interactions  between  valid  federates. 

Tests  may  identify  the  source  conditions  of  possible  interoperability-related  anoma¬ 
lies:  including  functional  compositions  and  manifold  representations  [HYOla].  Functional 
compositions  occur  when  the  computation  of  one  or  more  object  states  in  one  federate  de¬ 
pend  upon  the  data  provided  by  another  federate  and  violates  one  or  more  of  the  following 
interoperability  criteria  adversely  affecting  M&S  quality: 

•  Dependency  representation, 

•  Representational  accuracy, 

•  Range  consistency, 

•  Sensitivity  consistency, 

•  Temporal  representation, 

•  Internal  sensitivity, 

•  Error  consistency, 

•  Stochastic  consistency  [HYOla], 

Manifold  representations,  normally  exhibited  under  very  specific  conditions,  occur 
when  two  or  more  federates:  1)  represent  the  same  behavior  or  state,  and  2)  interact  either 
directly  or  indirectly.  Both  parallel  manifold  representations  and  sequential  manifold  rep- 


104 

This  phenomenon  is  common  to  chaos  theory,  and  is  also  known  as  a  sensitive  dependence  on  initial  conditions,  first  identified  by 
Edward  Lorenz,  a  meteorologist,  in  1960. 

Event  phase  anomalies  exhibit  a  timing  or  phase  error.  Event  ordering  anomalies  occur  when  the  simulated  object  produces  the  same 
events  as  the  simuland  under  identical  conditions,  but  in  a  different  order.  State  error  anomalies  are  identified  by  a  difference  between 
the  state  a  simulated  object  assumes  and  the  state  that  object’s  referent  exhibits  under  identical  conditions  and  the  difference  is  beyond 
tolerance  levels  [HYOla]. 
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resentations  may  cause  interoperability  problems,  including  aggregation  or  disaggregation 
issues  if  they  fail  to  meet  manifold  representation  interoperability  criteria  including:  state 
correspondence,  abstraction  transform,  or  state  continuity.  The  simulation  conceptual 
model106  provides  the  necessary  supporting  information  to  support  these  tests  [PacOla], 
while  the  Federation  Development  and  Execution  Processes  (FEDEP)  [GYL03]  include 
conceptual  model  development  procedures  and  guidelines  for  building  a  federation,  sup¬ 
porting  credible  verification  and  validation  [DSB+99]. 

The  level  to  which  heterogeneous  system  representation  anomalies  affect  the  credi¬ 
bility  of  a  federation  is  derived  from  the  representational  acceptability  criteria,  based  on  the 
federation’s  purpose  and  the  capabilities  the  federation  or  federate  must  meet  to  support 
that  purpose  [YPH01].  The  DoD  Modeling  and  Simulation  Verification,  Validation  Ac¬ 
creditation  Instructions  [DoD96]  supports  credible  M&S  development  and  includes  ac¬ 
ceptability  criteria107,  linking  the  roles  and  responsibilities  of  federation  life-cycle  devel¬ 
opment  and  operation  with  improved  decision-makers  confidence  in  the  results  when: 

•  Developers  design  and  implement  the  software  functionality  to  satisfy  the  criteria, 

•  V&V  agents  evaluate  federation  capabilities  against  the  criteria, 

•  Accreditation  agents  make  accreditation  recommendations  based  upon  the  criteria 

[YPH01], 

Department  decision-makers  accredit  the  use  of  the  federation  for  a  specific  purpose 
based  on  confidence  that  the  acceptability  criteria  were  credibly  established  and  imple¬ 
mented.  A  contributing  factor  to  the  substantive  interoperability  problem  has  been  an  in¬ 
consistent  implementation  of  the  Department’s  verification,  validation,  and  accreditation 
(VV&A)  policy  identified  in  Chapter  III.  The  Department’s  VV&A  processes  depend  on 
consistent,  credible  conceptual  models.  However,  Department’s  conceptual  model  imple¬ 
mentation  is  inconsistent,  and  the  theoretical  debate  over  conceptual  model  fonnats,  meth¬ 
ods,  uses  and  development  processes  continue  unabated.  In  addition  data  to  validate  De¬ 
partment’s  M&S  remains  elusive,  expensive  and  difficult  to  acquire,  adding  to  the  chal¬ 
lenges  of  conducting  creditable  VV&A. 


106  The  simulation  conceptual  model  is  the  developer’s  translation  of  modeling  requirements  into  a  detailed  design  framework,  from 

which  the  software,  hardware,  networks  and  systems/equipment  that  make  up  the  M&S  can  be  built  [HYOla]. 

107 

Acceptability  criteria  should  be  necessary  for  the  purpose,  specific,  measurable  and  complete  [YPH01] 


90 


Substantive  interoperability  issues  also  exist  due  to  the  level  of  software  engineer¬ 
ing  discipline  and  the  general  lack  of  process  maturity  exhibited  by  Department  M&S 
software  developers  (see  Chapter  VII).  The  Department  initiated  major  new  large-scale 
M&S  development  efforts,  JSIMS,  JWARS,  and  JMASS,  to  replace  many  of  the  aging  leg¬ 
acy  systems  and  decrease  the  issues  with  substantive  interoperability.  However,  all  three 
major  systems  are  well  behind  schedule,  significantly  over-cost,  and  failed  to  demonstrate 
they  meet  the  original  requirements.  The  development  and  implementation  of  software  ar¬ 
chitectures,  including  product  lines,  in  the  Department  while  still  very  immature  as  it  mir¬ 
rors  progress  made  in  the  private  sector,  has  the  potential  to  improve  the  Department  M&S 
substantive  interoperability  problem  in  the  future.  Compounding  the  VV&A  deficiencies  is 
the  use  of  over-subscribed  terms  for  M&S  quality  attributes  such  as  fidelity. 

F.  FIDELITY  HETEROGENITY 

Chapter  VI  discusses  M&S  quality  attributes,  including  fidelity,  and  Table  6-2  pro¬ 
vides  the  current  Department  context  for  M&S  fidelity.  However,  continued  concern  about 
M&S  fidelity  within  the  SISO  led  to  establishment  of  two  fidelity  study  groups  in  the  late- 
1990s  with  charters  to  produce  a  fidelity  conceptual  framework  for  M&S  fidelity  and  a 
glossary  of  fidelity  related  terms  [Gro99].  The  Simulation  Standards  Interoperability  Or¬ 
ganization  chartered  the  Fidelity  Definition  and  Metrics  Implementation  Study  Group 
(ISG-FDM)  [Gro99]  and  a  follow-on  effort,  the  Fidelity  Experimentation  ISG  (ISG-FEX) 
[RGH00]  to  address  the  issue  of  fidelity.  These  efforts  produced  collaborative  integrated 
project  reports  addressing  appropriate  M&S  fidelity  definitions,  a  fidelity  conceptual 
framework,  fonnulas  and  concepts  for  the  SISO  community,  and  recommendations  for  fu¬ 
ture  DMSO  M&S  efforts.  However,  fidelity,  the  subject  of  many  scientific  papers,  studies, 
analyses,  and  reports,  still  lacks  an  acceptable,  quantifiable  methodology. 

The  [Gro99,  RGH00]  reports,  a  major  SISO  collaborative  effort  by  over  one  hun¬ 
dred  researchers,  provided  a  summary  of  workable  contextual  frameworks  within  which 
M&S  fidelity  is  considered,  with  an  explicit  indication  of  how  such  frameworks  would  re¬ 
late  to  the  larger  theoretical  context  of  modeling  and  M&S  theory,  including  initial  identifi¬ 
cation  of  methods,  approaches,  and  metrics  for  usefully  defining,  estimating,  and  measur- 
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ing  aspects  of  M&S  fidelity.  Fidelity  of  a  model  or  simulation  defines  the  terms  of  the 
relevant  referent  and  the  capabilities  of  the  model  or  simulation.  Fidelity  also  describes  the 
essential  characteristics  of  the  model  or  simulation  relative  to  its  referent. 

[Gro99,  RGHOO]  also  identified  the  dimensions  and  attributes  of  M&S  fidelity 
based  on  the  characteristics  of  reality,  addressed  by  entities,  factors  and  relationships 
within  the  simulation’s  enumeration  of  the  problem’s  critical  aspects.  [Gro99,  RGHOO] 
further  defined  the  scope  of  involved  entities  and  the  identifiable  depths  of  entities. 

[Gro99]  identified  three  dimensions  of  simulation  fidelity,  which  include  a  number  of  the 
possibilities,  or  a  percentage  of  factors 

The  first  dimension  of  simulation  fidelity  [Gro99]  identified  is  enumeration  of  the 
involved  entities,  addressed  in  both  scope108  and  depth109.  In  the  second  dimension  of 
simulation  fidelity  [Gro99]  included  the  identification  of  involved  factors110  relating  the 
processes,  which  influence,  impact,  or  describe  entity  states  and  behavior.  Specification  of 
the  significant  relationships  among  entities  is  the  third  dimension  of  simulation  fidelity 
forwarded  by  [Gro99]  to  explicitly  articulate  the  assumptions  about  dependencies  or  inde¬ 
pendence  among  entities.  [Gro99]  also  describes  dependent  relationships  not  explicitly 
identified  as  “otherwise”111. 

Fidelity  presents  significant  challenges  for  accurately  addressing  the  size,  scope, 
and  complexity  of  the  real  world  and  our  current  understanding  of  the  real  world  preclude 
its  use  as  a  practical  measure.  Since  the  real  world  is  not  a  good  foundation  to  measure  fi¬ 
delity,  research  efforts  have  identified  the  need  for  common  referent,  a  commonly  under¬ 
stood  standard  [Gro99,  RGHOO],  This  raises  the  point  of  how  to  assess  the  fidelity  of  a 
simulation  against  only  those  aspects  of  the  referent  needed  to  support  the  simulation;  oth- 
erwise,  if  the  simulation  represented  all  aspects  of  the  simuland  '  it  would  in  fact  be  the 
simuland  [Gro99,  RGHOO]. 


Scope  addresses  the  spectrum  of  entities  represented  by  the  simulation  [Gro99]. 

109 

Depth  is  the  level  at  which  entities  in  a  simulation  can  be  individually  identified  [Gro99] 

Factors  may  include  components,  materials,  internal  parameters,  algorithms,  and  parameters  related  to  measures  of  performance 
(MOPS)/effectiveness  (MOE)/  merit  (MOM)  [Gro99]. 

The  “otherwise”  category  described  by  [Gro99]  as  a  description  for  all  dependent  relationships  that  cannot  be  described  more  pre¬ 
cisely. 
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A  simuland  is  the  system  being  simulated  by  a  simulation  [DoD98]. 
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The  second  obstacle  [Gro99,  RGHOO]  identified  is  the  specification  of  the  simula¬ 
tion.  Defined  simulation  fidelity  has  the  potential  to  act  as  a  metric  for  describing  how  well 
the  behavior  of  the  unique  aspects  of  the  simulation  matches  selected  parts  of  the  simuland. 
[Gro99,  RGHOO]  suggests  the  simulation  also  has  many  other  characteristics  describing  the 
nature,  behavior  and  character  of  the  simulation,  independent  of  the  simuland,  requiring  a 
complex,  multidimensional  set  of  measures.  The  multidimensional  set  of  measures  in¬ 
clude,  but  is  not  limited  to,  the  simulation’s  intrinsic  and  extrinsic  quality;  development 
costs;  resources  needed  to  develop  scenarios  and  execute  the  simulation;  the  extent  that  de¬ 
composition,  aggregation  and  interfaces  must  be  described;  and  the  intended  computation 
environment  [Gro99,  RGHOO], 

Simulation  fidelity  attributes  address  the  quality  of  the  parameters  within  the  di¬ 
mensions  of  M&S  fidelity  and  include  the  following  characteristics:  factor  order,  accuracy, 
precision,  timeliness,  consistency,  repeatability,  and  possible  error  sources  [Gro99, 
RGHOO].  Generally,  within  simulations  the  higher  the  factor  order113,  [Gro99,  RGHOO] 
suggests  the  greater  the  fidelity.  However,  since  factors  may  be  high  order,  low  order,  or 
some  other  order  within  the  simulation,  [Gro99,  RGHOO]  suggests  terms  of  factors  provide 
better  definition  of  simulation  fidelity  in,  in  lieu  of  using  the  concept  of  aggregation.  Other 
simulation  fidelity  attributes  include  consistency114,  repeatability115,  and  people  issues116. 

A  closely  related  concept  including  aspects  of  abstraction  and  aggregation  is  multiresolu¬ 
tion  modeling. 


113 

The  quality  of  parameters  within  a  simulation  start  with  the  order  of  the  parameter  description  of  the  factor  in  the  simulation  with  a 
zero  order  parameter  description  being  a  constant  value,  a  first  order  parameter  description  would  vary  in  one  way  according  to  some 
statistical  distribution,  a  second  order  parameter  description  would  very  in  two  ways,  etc  [Gro99,  RGHOO] 

114 

Consistency  addresses  whether  simulation  results  are  biased  and  stable  in  terms  of  the  dispersion  of  results  created  by  the  simulation 
processes  [Gro99,  RGHOO]. 

Repeatability  means  that  a  simulation  should  produce  the  same  results  given  the  same  stimuli  [Gro99,  RGHOO]. 

People  issues,  often  ignored,  is  a  special  fidelity  concern  since  as  players,  operators  or  analysts  can  impact  accuracy,  precision,  con¬ 
sistency,  and  repeatability  in  simulation  results  [Gro99,  RGHOO]. 
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G.  MULTIRESOLUTION  MODELING  (MRM)  HETEROGENITY 

[Zei91,  Zei92a,  Zei92b,  Dav92a,  Edd92,]  introduced  concepts  supporting  variable- 
resolution  modeling  (VRM),  a  precursor  to  multiresolution  methodology  (MRM)  .  These 
early  citations  provided  definitions,  introduced  basic  concepts,  identified  types  of  VRM, 
provided  examples,  illustrated  the  importance  of  VRM  for  integrating  new  models  or  redes¬ 
igning  existing  models,  including  potential  cost  savings  [Sil92].  [Dav92a,  Dav93]  also  dis¬ 
cussed  three  classes  of  variable  resolution  they  termed:  selected  viewing,  alternative  sub¬ 
models  (or  model  families),  and  integrated  hierarchical  variable  resolution  (IHVR).  The 
IHVR  concept  forwarded  [Dav92a,  Dav93]  by  defines  critical  processes,  hierarchically 
composed  of  subordinate  processes. 

The  study  effort  also  built  on  previous  MRM  research  by  [Ham96,  HNP97,  Dav98, 
DB99,  SHBOO,  and  DB03],  which  proved  invaluable  for  establishing  a  foundation  support¬ 
ing  the  introduction  of  software  architecture -based  M&S  product  lines.  [Dav98]  notes  that 
MRM  is  “still  a  frontier  subject  in  modeling  and  simulation”  [Dav98],  with  more  work 
needed  in  diverse  areas  including  fundamental  theory,  computational  tools,  and  visualiza¬ 
tion  techniques.  Important  theoretical  predecessor  works  include  object-oriented  modeling 
and  discrete-event  simulation  concepts  by  [Zei91],  modular  hierarchical  model  representa¬ 
tions  by  [Zei92a],  a  review  of  stochastic  versus  deterministic  methodologies  in  combat 
simulations  by  [LucOO],  and  the  development  of  methodologies  for  structured  families  with 
multiple  levels  of  resolution  [Zei92b]. 

Furthermore,  [DB99]  recognized  the  limitations  of  programming  and  operations  re¬ 
search  and  identified  a  need  for  in-depth  research  “that  leads  to  sound  theories  and  designs 
well  before  coding  even  begins,”  [DB99]  and  “should  reflect  sound  principles  of  software 
engineering,  including  the  ability  to  design  in  an  object-oriented  modeling  framework” 
[Dav98,  DB99].  In  lieu  of  combining  several  different  models  at  different  levels  of  resolu¬ 
tion  into  a  federation  or  other  form  of  distributed  simulation  operation,  [Ham96,  HNP97] 


Some  of  the  many  terms  that  MRM  issues  arise  under  are  model  abstraction  theory,  problem  decomposition,  variable-resolution 
modeling,  variable-fidelity  modeling,  hierarchical  modeling,  aggregation  and  disaggregation,  chunking,  and  building  of  lumped,  reduced, 
or  simplified  models  [Dav98]. 

[Zei92b]  suggests  that  resolution  is  the  degree  to  which  objects  and  processes  in  the  real  world  are  resolvable  in  the  model,  and  fur¬ 
thermore,  the  resolution  level  is  synonymous  with  the  level  of  detail  [Zei92b] . 
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discuss  an  alternative  strategy  of  developing  and  operating  at  multiple  levels  of  resolution. 
[DBW87,  BM92,  Dav92a,  DH92a,  DH92b,  Dav93,  Ham96,  DC97,  HNP97,  Dav98,  DB99, 
MDBOOA,  MDBOOb,  DavOO,  DBMOO,  BDMOO,  and  AdeOl]  developed  a  significant  body 
of  knowledge  on  multiresolution  modeling  and  multi-resolution  multi-perspective  model¬ 
ing  (MRMPM). 

[DB99]  further  defines  multiresolution  modeling119.  [DH92a]  noted  that  achieving 
interoperability  in  diverse,  independently  developed  M&S  with  different,  but  overlapping 
resolutions,  based  on  different  assumptions,  limitations,  and  perspectives  is  a  major  chal¬ 
lenge  in  contemporary  M&S.  [Dav92a,  Dav93  and  DBM99]  in  Figure  4-1  illustrate  that 
the  term  resolution  represents  many  different  concepts  and  dimensions. 

However,  [DavOO]  noted  MRM  may  not  be  sufficient  for  applications  requiring  dif¬ 
ferent  abstractions  or  perspectives,  that  vary  by  conception  of  the  system  or  use  of  vari¬ 
ables,  and  requires  a  new  model  construct  for  both  multiple  resolution  and  multiple  per¬ 
spectives  (MRMPM).  In  [MDBOOa]  models  and  families  of  models  employed  hierarchical 
trees  to  remain  mutually  consistent  across  multiple  resolution  and  perspectives.  [BDMOO] 
employed  an  exploratory  analysis  technique  using  low-resolution  models  at  the  macro  level 
of  analysis  and  high-resolution  models  to  support  validation  efforts. 

Adding  to  the  existing  body  of  knowledge,  [DavOO,  DBMOO,  BDMOO,  and  Dav93, 
and  HOB95]  compiled  a  significant  body  of  work  in  multi-resolution,  multi-perspective 
modeling  (MRMPM)  complementing  the  existing  aggregation  /  disaggregation  research. 
Taking  a  hybrid  top-down  /bottom-up  approach  to  validation,  [Dav95a]  proposed  a  family 
of  models  optimally  designed  top-down  and  bottom-up  concurrently,  noting  that  the  more 
detailed  models  may  only  have  selected  details  and  may  lack  the  infonnation  required  to 
develop  valid  aggregate  models.  [HNP97]  suggest  the  development  of  a  resolution  hierar¬ 
chy  supporting  the  ability  to  control  the  bounds  of  inputs.  [Dav95a]  further  indicates  ag¬ 
gregated  models  may  be  able  to  establish  the  context  and  boundary  conditions  for  events 
required  in  models  that  are  more  detailed.  [Dav95b]  applied  the  combat  ratio  3:1  rule  to 


Building  a  single  model,  a  family  of  models,  or  both  to  describe  the  same  phenomena  at  different  levels  of  resolution  [DB99]. 
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In  combat  operations  the  ratio  of  attackers  to  defenders  for  a  successful  offensive  operation  [Dav95b] . 
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the  complexities  of  aggregation  and  disaggregation  and  detailed  how  to  achieve  reasonably 
accurate  and  higher-aggregated  levels. 

[TNM01]  later  addressed  the  multi-resolution  question  in  federations,  and  identified 
the  problems  that  arise  from  federations  or  interaction  between  simulations  with  different 
levels  of  detail  or  abstraction  requiring  aggregation  and  disaggregation  of  information  re¬ 
quired  by  an  HLA  federation.  [TNM01]  submit  two  new  approaches:  the  middleware,  and 
the  federate-based  approaches  to  replace  the  module-based  approach  to  implement 
aggregation/disaggregation  in  HLA  environments. 

Lacking  integration,  most  model  hierarchies  are  just  a  collection  of  models  with  dif¬ 
ferent  resolution  and  some  type  of  procedures  for  calibrating  selected  input  parameters  of 
the  lower-resolution  with  higher  resolution  models.  [Dav98]  noted  the  different  design  phi¬ 
losophy  between  the  normal  bottoms-up  approach  favored  by  the  Department,  which  as¬ 
sumes  truth  is  inherent  in  the  most  detailed  models,  whereas  MRM,  as  noted  in  Table  4-1 
supports  any  approach  in  achieving  mutually  calibrated  models  in  a  family.  Today,  with 
the  end  of  the  Cold  War  added  increased  uncertainty  and  added  significant  complexity  to 
possible  analytical  issues  and  M&S  development  arising  from  the  increasing  instability, 
new  variables,  and  unfamiliar  post-Cold  War  geographical-political-military  environments. 
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Figure  4- 1 .  Resolution  Dimensions  (From  [Dav98]) 

Different  design  methodologies  listed  in  Figure  4-2,  supported  by  several  MRM 
tools  exist.  [DBMOO]  described  the  implementation  of  a  fast-running,  stochastic,  mul- 
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tiresolution,  stochastic  model  PGM121  Effectiveness  Modifier  (PEM)  for  exploratory  analy¬ 
sis  into  the  complex  situational  and  tactical  factors  of  long-range  precision  fires.  In  another 
effort  to  conduct  insight-oriented  exploratory  analysis,  [MDBOOb]  described  the  effort  to 
employ  the  PC-based  EXHALT  model,  based  on  the  analytical  methods  identified  in 
[DC97],  to  conduct  a  study  of  interdiction  capabilities  to  quickly  stop  an  invading  anny. 
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GENE  RALITY 
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SIG  N  ALTERNATIVEUSER  MODES  AND  CORESPOND¬ 
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SIMILAR  ENTITIES  (E.G.,  SHOOTERS,  WHETHER  AIR¬ 
CRAFT  OR  M  ISSILES. 

SEEK  APPROXIMATIONS  THAT  SIMPLIFY  MODEL 
STRUCTURE  SENSIBLY,  TH  EREBY  INCREASING 
ANALYTICAL  FLEXIBILITY  FOR  EXPLORATION  AND 
QUICK  CHANGES  OF  ASSUMPTIONS. 


•  FOCUS  ON  SIM  ULTION  RATHER  THAN 
SENSITIVITY 


CHOOSE  PERSPECTIVE  UP  FRONT  FROM  POSSIBLE 
MODULARIZED  COMBINATIONS  OF  AGGREGATE 
VARIABLES. 


FOCUS  ON  SIM  ULATIONRATII  ER  THAN 
NEEDS  FOR  SENSITIVIYY  PROGRAM 


INSERT  MULTIPLIERS,  STRETCHERS,  AND  OTHER 
‘ABSTRACTIONS  OF  CONVENIENCE'. 


•DESIGN  WITH  BOTTOM-UP  ETHIC 


DESIGN  WITH  ETHIC  OF  USING  BOTTOM-UP, 
TOP -DOWN,  AND  SIDEWAYS  APPROACHES  IN 
MUTUALLY  SUPPORT  IV  E  WAYS. 


Figure  4-2.  Distinction  between  Normal  and  MRM  Design  (From  [Dav98]) 


In  a  related  effort,  the  ARP  A/Tri-Services  sponsored  the  Rapid-prototyping  of  Ap¬ 
plication  Specific  Signal  Processors  (RASSP)  program  research  by  [HMB+00]  reviewed 
and  compared  simulation  terminology,  interoperability  challenges,  and  previous  modeling 
taxonomies;  as  a  basis  for  designing  a  multi-axis  taxonomy  to  describe  the  information 
content  of  computer  model  types;  and  define  abstraction  levels  supporting  development  of 
interoperable  models.  The  RASSP  modeling  taxonomy  provides  a  framework  to  categorize 
models  based  on  a  set  of  attributes,  which  may  distinguish  models  for  different  intended 
uses,  and  establish  formal,  concise,  unambiguous  definitions  for  various  model  types  which 

•  Represent  model  attributes  relevant  to  model  designer  and  users, 

•  Provide  a  readily  understandable  common  terminology, 

•  Identifies  five  distinct  model  characteristics:  temporal  detail,  data  value  detail,  func¬ 
tional  detail,  structural  detail,  and  programming  level, 
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PGM  is  an  acronym  for  precision-guided  munitions 
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•  Consists  of  a  set  of  attributes  characterizing  a  model’s  relative  resolution  of  details 
for  important  model  aspects  shown  in  Figure  4-3  [HMB+00]. 
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Figure  4-3.  RASSP  Taxonomy  Model  (From  [HMB+00]) 

Within  the  RASSP  Taxonomy  Model  at  Figure  4-3,  a  given  model  instance  de¬ 
scribes  information,  presented  graphically  at  one  specific  level  within  the  model,  even 
though  some  terms  may  span  a  range  of  abstraction  levels  .  The  RASSP  Taxonomy 
Model  provides  five  categories  of  resolution  in  which: 

•  Temporal  Resolution  Axis  represents  the  resolution  of  events  presented  on  a  time 
scale, 

•  Data  Resolution  Axis  represents  the  resolution  of  the  format  of  values  specified  in 
the  model, 

•  Functional  Resolution  Axis  represents  the  level  of  detail  of  a  model  describing  the 
functionality  of  a  component  or  system, 


122 

The  abstraction  is  an  indication  of  the  level  of  detail  specified  about  how  a  function  is  to  be  implemented  and  is  adversely  related  to 
the  level  of  detail.  If  there  is  much  detail,  or  high  resolution,  the  abstraction  is  considered  low,  and  the  abstraction  levels  form  a  hierar¬ 
chy  [HMB+00]. 
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•  Structural  Resolution  Axis  represents  the  level  of  detail  of  a  model  describing  how 
a  component  is  constructed  from  its  constituent  part, 

•  Software  Programming  Resolution  Axis  represents  the  level  of  granularity  of  soft¬ 
ware  instructions  that  the  model  of  a  hardware  component  interpret  when  executing 
target  software  [HMB+00] . 

The  level  of  abstraction  does  not  indicate  accuracy;  in  the  same  way  precision  is 
different  from  accuracy,  since  two  different  models  of  a  given  function  with  different  ab¬ 
straction  may  be  equally  as  accurate.  In  addition  the  RASSP  Taxonomy  Model  provides 
two  views  of  the  attributes,  the  internal  view  as  viewed  from  inside  the  model,  and  the  ex¬ 
ternal  view  as  viewed  from  the  model’s  interface  boundary,  each  view  supporting  the  tem¬ 
poral,  value,  structure,  and  function  resolutions  for  a  total  of  eight  attributes  enabling  clar¬ 
ity  and  precision,  unlike  many  other  taxonomies  which  may  mix  attributes  [HMB+00] . 

The  internal  resolution  references  how  a  model  describes  the  timing  of  events,  functions, 
values,  and  structures  of  the  elements  contained  within  the  models  boundary,  whereas  the 
external  resolution  describes  how  a  model  defines  the  interface  of  the  modeled  device  to 
other  devices  [HMB+00] . 

The  RASSP  Taxonomy  Model  supports  several  hierarchies123:  the  functional  or 
logical  hierarchy,  and  the  structural  of  physical  hierarchy.  The  functional  hierarchy  de¬ 
composes  a  system  according  to  functions  (e.g.,  receiver,  register,  transmitter),  and  the 
structural  hierarchy  decomposes  a  system  according  to  physical  structure  (e.g.,  frames, 
racks,  chassis)  or  logical  relationships  (e.g.,  data  structures)  [HMB+00]. 

Newer  MRM  theories  include  possible  modifications  to  the  HLA  RTI  and  possible 
use  of  components.  [SKO+Ol]  completed  additional  work  on  MRM  methods  and  proposed 
the  use  of  state  variables  of  simulation  objects  or  environments  in  an  MRM  simulation  sys¬ 
tem  to  detennine  the  resolution  of  the  models.  [AS01]  submitted  a  new  HLA  service,  the 
multiresolution  management  service,  using  existing  HLA  services  as  a  suggested  addition 
to  the  RTI.  [HBM96]  proposes  building  multiple  levels  of  resolution  with  defined  ranges, 
into  multiple  independent  models,  as  opposed  to  one  supennodel,  with  the  ability  to  re¬ 
move  representations,  which  are  not  required  to  complete  comparative  analysis.  In  a  simi- 


123 

Hierarchies  in  RASSP  are  a  multi-level  system  supporting  aggregation  and  decomposition,  in  which  a  node  at  a  given  level  of  the 
hierarchy  may  be  represented  by  the  set  of  its  descendent  nodes  on  the  next  lower  level  [HMB+00]. 
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lar  concept,  [CGG98]  explained  the  JSIMS  Military  Modeling  Framework  initiative,  a  De¬ 
partment  enterprise-wide  Tiger  Team  initiative  to  develop  Mission  Space  Objects  formed 
with  interoperable,  multiresolution  models  from  reusable  and  dynamically  re-configurable 
components  to  achieve  a  common  interpretation  within  a  synthetic  battle  space. 

H.  INTEROPERABILITY  AND  HETEROGENITY  IN  THE  C4ISR  DOMAIN 

Interoperability  poses  significant  challenges  in  many  areas  including  the  Command, 
Control,  Computers,  Communications,  Intelligence,  Surveillance,  and  Reconnaissance 
(C4ISR)  dimension.  [DoD02d]  defines  critical  Department  M&S  requirements  for  im¬ 
proved  C4ISR.  However,  there  are  currently  severe  M&S-to-C4ISR  shortcomings  identi¬ 
fied  by  [SFJ+98,  RHS99,  Sut99,  Tol99,  DAOO,  TLW+00,  ML01,  BBN+01,  HMT+01, 
WHL01,  CWS02,  DoD02c,  DoD02d,  ST02]  for  modeling  information  superiority. 
[DoD02d]  summarizes  these  shortcomings  and  notes: 

•  Good  information  superiority  models  do  not  exist, 

•  Pre-scripted  battlefield  entities  are  non-responsive  and  lower  study  fidelity, 

•  Existing  models  are  simplistic,  speculative,  and  too  narrowly  focused,  lacking  the 
ability  to  do  end-to-end  analysis, 

•  Current  force-on-force  models  do  not  account  for  C4ISR  or  make  broad  assump¬ 
tions, 

•  A  lack  of  system  performance  data, 

•  The  Department  needs  integrated  treatment  of  threat  signatures  and  sensor  mission 
management, 

•  Suitable  Opposing  Force  (OPFOR)  representations  are  lacking  [DoD02d]. 

A  SISO  M&S-to-C4I  Interoperability  Working  Group  report  [TLW+00]  identified 
that  uncoordinated  M&S  and  JTA  standards  have  been,  and  continue  to  be  developed  by 
both  communities,  and  several  factors  inhibit  a  completely  seamless  M&S-to-C4ISR  inter¬ 
face  including: 

•  Applicable  standards  (HLA,  JTA)  in  both  domains  continue  to  evolve  quickly, 

•  Interoperability  between  C4ISR  systems  or  components  may  span  multiple  levels, 

•  Each  domain  must  be  able  to  retain  authority  and  responsibility  for  data  V&V  and 
security, 

•  Each  domain  has  access  point  responsibilities  for  data  management,  metadata,  and 
conversions, 

•  Both  domains  must  support  active  interfaces  at  the  domain  boundaries, 
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•  Both  architectures  require  a  translation  capability  until  future  architectures  employ 

the  same  data  elements  based  on  a  common  design  [TLW+00]. 

The  C4ISR  Architecture  Framework  [AWG97]  provides  the  strategic  direction  for 
C4ISR  architecture  development  throughout  the  Department,  and  provides  guidance  on  the 
development  of  a  broad  set  of  products  used  to  document  the  operational,  system  and  tech¬ 
nical  architecture  views.  An  M&S-to-C4ISR  interoperability  framework  proposed  by 
[TLW+00]  includes  a  three-phase  plan  with  near-,  mid-,  and  long-tenn  solutions  evolving 
from  a  limited  message-based  as-is  architecture.  The  to-be  architecture  shares  common 
databases,  implicit  interoperability,  “plug  and  play”  capabilities,  and  full  duplex  informa¬ 
tion  exchange  [TLW+00] . 

[TLW+00]  identifies  success  in  the  mid-term  architecture  with  common  compo¬ 
nents,  improved  interfaces,  and  common  standards,  with  simulations  updating  C4I  systems 
over  two-way  communications  initialized  by  either  the  simulation  or  the  C4I  system.  Cur¬ 
rently  translators  establish  this  interface,  although  translators  represent  an  interim,  expen¬ 
sive,  error-prone,  single-point-of-failure  software  solution  until  common,  credible,  interop¬ 
erable  components  become  available.  [TLW+00]  proposes  solutions  including  a  C4I/M&S 
technical  reference  model  (TRM),  a  broad  data  class  (metadata),  a  reference  FOM,  and 
SISO  guides  to  linking  C4I  to  M&S.  The  Levels  of  Information  Systems  Interoperability 
(LISI)  model  [AWG98],  shown  in  Figure  4-4  illustrates  components  or  systems  spanning 
multiple  levels  of  sophistication,  supporting  various  system-to-system  infonnation  ex¬ 
changes. 

Current  department  data  quality  initiatives  support  the  object-oriented  approach. 
The  U.S.  Army  developed  the  Army  Object  Model  Standards  Category  (OMSC)  [JHB98, 
HB98a.  HB98b,  MG98]  for  M&S  domain  objects,  and  the  Army  Integrated  Core  Data 
Model  (AICDM)  a  standard  reference  data  model  [WHL+01],  which  includes  the  Joint 
Common  Data  Base  (JCDB),  the  Army  Land  C2  Infonnation  Exchange  Data  Model,  and 
the  C2  Core  Data  Model  [MHL+02],  [HB99,  LFWOO,  WHL+01,  WHL+02]  assessed  in¬ 
teroperability  between  M&S  and  C4ISR  data  models,  and  evaluated  the  degree  of  align¬ 
ment  to  determine  common/data  object  interoperability. 
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An  Anny  M&S  Interoperability  Working  Group  [BunOO],  reviewed  the  technical 
challenges,  complexity,  cost,  and  criticality  of  M&S  interoperability,  and  provided  a  defini¬ 
tion  for  M&S  interoperability124  and  proposed  four  types  of  interoperability  within  a  ge¬ 
neric  3-tier  architecture  for  simulations  including: 

•  HLA  Federation  supporting  federation  runtime  interoperability, 

•  Scenario  data  sharing  among  different  M&S  (e.g.,  environment,  weapons  effective¬ 
ness  data)  including  automated  tools,  and  reduced  cost, 

•  Pedigreed  data  supporting  the  requirement  for  a  lower-level  M&S  (e.g.,  higher  reso¬ 
lution,  precision)  to  develop  data  for  a  higher-level  M&S  in  the  Department’s  M&S 
hierarchy, 

•  Model  /  Algorithm  sharing  in  M&S  with  different  architectures  and  fidelity  re¬ 
quirements  [BunOO], 

Related  initiatives  include  the  Department’s  Data  Architecture  [JTA02a],  interfac¬ 
ing  with  C4I  systems  [TolOl],  the  development  of  a  DII  COE  M&S  infrastructure  [HSOOa], 
interoperability  certification  [BKW+02],  object  architectures  supporting  semantically  in¬ 
teroperable  systems  [BCC+00],  and  weapons  effectiveness  [LPA+02].  [TH03]  noted  major 
discrepancies  between  the  current  OMSC  and  the  JCDB  standard  reference  data  models 
with  implications  that  the  simulations  currently  under  development  do  not  support  the 
Army  Battle  Command  Systems,  and  require  custom  interface  software  or  translators. 

A  follow-on  SISO  M&S  /  C4I  study  group  [BCC+02]  confirmed  the  need  for  a 
common  technical  reference  model  (TRM)  [TLWOO]  to  improve  interoperability  based  on  a 
review  of  M&S  reference  models,  a  prototype  C4I  /  M&S  interoperability  TRM,  in  form  a- 
tion  exchange  activities,  a  general  unified  model  (GUM),  and  related  reference  models. 
Proposed  future  initiatives  included  continued  analysis  of  interoperability  theories,  a  C4I  / 
M&S  TRM  use-case  study  and  development  of  TRM  user  guides.  [TH03]  proposed  addi¬ 
tional  challenges  to  the  current  C4ISR-to-M&S  incompatibility  issue  with  six  theses  he 
framed  as  grand-challenges. 

In  addition,  [TH03]  propose  to  foster  the  development  of  information  techniques 
and  systems  to  support  the  war  fighter  based  on  the  same  views  of  the  world,  a  common 
ontology,  a  common  technical  framework,  and  a  common  understanding  of  the  constituent 

124 

M&S  interoperability  is  the  ability  of  a  model  or  simulation  to  provide  services  from  other  models  and  simulations,  and  to  use  the 
services  so  exchanged  to  enable  tern  to  operate  effectively  together  [BunOO]. 
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dynamics.  [TH03]  also  express  concern  with  the  perception  that  the  M&S  community  has 
not  provided  enough  emphasis  on  the  interoperability  issues  between  C4ISR  systems  and 
M&S,  beyond  the  development  of  representations  of  units,  equipment,  and  behavior. 

As  a  result,  [TH03]  identified  a  need  for  a  common  architecture,  a  common  ontol¬ 
ogy,  a  common  set  of  algorithms  and  methods  supporting  a  common  overarching  concept 
bridging  the  different  methodologies.  The  Department’s  alleged  M&S  research  void  cited 
by  [TH03]  provides  a  major  impetus  for  this  dissertation  and  the  development  of  product 
line  domain  architecture.  The  six  theses  proposed  by  [TH03]  to  improve  the  C4ISR  sys¬ 
tems  and  military  simulation  systems  ability  to  support  future  military  operations  include: 

•  Complementary  techniques  needed  for  future  military  operations, 

•  Use  of  the  same  commercial  standards, 

•  Use  of  the  same  common  architecture, 

•  Use  of  data  and  object  models  based  on  the  same  common  ontology, 

•  Use  of  the  same  common  set  of  algorithms  and  methods, 

•  A  common  overarching  concept  for  C4ISR  systems,  and  military  simulation  sys¬ 
tems  development  and  evolution  [TH03]. 

Finally,  [PacOla]  contends  that  although  the  Joint  Technical  Architecture  (JTA)  provides 
the  Department  architecture  for  exchanging  information  at  the  current  time,  and  the  HLA 
defines  interoperability  standards  for  M&S,  no  standards  currently  exist  for: 

•  Decomposing  M&S  into  entities  and  processes, 
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•  Representation  abstraction  "  of  the  simulated  subject, 

•  Documenting  authoritative  representations  in  the  M&S  conceptual  model  [PacOla]. 

I.  SUMMARY 

Chapter  IV  reviewed  several  heterogeneity  challenges  to  Department  M&S  credibil¬ 
ity.  Improving  the  credibility  of  large-scale,  legacy  Department  M&S  requires  an  under¬ 
standing  of  the  underlying  systemic  causes  generating  the  pre-conditions  for  heterogeneous 
system  representation  anomalies,  especially  in  federation  interoperability.  These  diverse 
challenges  include  syntactic  and  semantic  heterogeneity,  data  heterogeneity,  complexity, 
interoperability,  technical  and  substantive  interoperability,  fidelity  heterogeneity,  and  mul¬ 
tiresolution  modeling  heterogeneity. 
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Abstraction  is  a  general  model  mapping  process  that  takes  a  base  model  into  a  lumped  mode  where  the  models  may  be  expressed  in 
different  formalisms,  and  subsumes  aggregation  as  a  special  case  [Zei92b]. 
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Authoritative  representations  have  been  consistently  identified  as  major  objectives 
in  Department  M&S,  yet  few  of  the  [DoD95,  DoDOla]  objectives  for  authoritative  repre¬ 
sentations  have  been  achieved  [MosOO,  CraOla,  CraOlb].  Senior  civilian  and  military  deci¬ 
sion-makers  have  significant  reservations  that  Department  M&S,  as  currently  practiced, 
can  meet  the  serious  future  demands  generated  by  national  security  requirements  for  M&S 
in  the  Department.  In  addition,  current  Department  M&S  users  still  require  accurate,  inter¬ 
operable  elements  of  the  mission-space  and  the  draft  [DoDOla]  retains  authoritative  repre¬ 
sentations  as  a  Department-wide  M&S  objective  including  U.S.,  allied,  friendly,  coalition, 
neutral  and  threat  and  paramilitary  system  representations  for  C4ISR,  combat,  combat  sup¬ 
port  and  combat  service  support  systems  and  processes.  Successful  attainment  of  these  in¬ 
teroperability  objectives  by  the  Department  and  international  M&S  communities  assumes 
the  acceptable  resolution  of  many  heterogeneity  issues,  including  abstractions,  networks, 
federations  and  representations. 

Abstraction  is  an  intrinsic  human  activity  employed  to  reduce  or  filter  the  full  com¬ 
plexity  of  the  real  world  to  manageable  proportions  by  deleting  unnecessary  detail  in  order 
to  provide  data  in  context  (e.g.,  infonnation)  supporting  the  decision-making  process 
[HJS97].  Abstraction  mechanisms  [Til98]  selectively  emphasize  the  level  of  detail,  em¬ 
phasizing  the  details  pertinent  to  the  intended  use,  while  hiding  unneeded  detail.  [Sow88] 
discussed  the  three  most  common  abstraction  mechanisms  classification,  aggregation,  and 
generalization  [Sow88,  Til98]. 

Data  models  establish  the  essential  properties  and  well-defined  relationships  be¬ 
tween  raw  data  in  a  system  in  a  form,  which  supports  efficient  storage,  timely  retrieval  of 
data,  and  provides  the  basis  for  tools  and  techniques  to  support  data  modeling.  A  data 
model  captures  the  static  and  dynamic  characteristics  supporting  data-related  processes, 
with  static  properties  defined  in  a  database  schema  and  the  dynamic  characteristics  devel¬ 
oped  into  specifications  for  reports,  queries,  and  transactions.  The  schema  maintains  the 
object  type  definitions,  attributes,  relationships  and  static  constraints  of  the  data  repository 
or  database,  an  instance  of  the  schema. 

The  most  significant  record-based  logical  data  models  include  the  hierarchical  data 
model  and  the  relational  data  model.  These  classic  highly  machine-oriented,  and  record- 
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oriented  data  models,  or  syntactic  data  models  are  well  suited  to  the  computer  environment 
and  organized  for  optimal  efficiency  supporting  storage  and  retrieval  operations,  however 
these  models  lack  semantic  relativism  and  abstraction  mechanisms  required  for  dealing 
with  complexity  in  large-scale  systems  [BCE+00]. 

[Til98]  describes  a  second  category,  semantic  data  models,  which  combine  basic 
knowledge  representation  techniques  with  database  technology,  and  represent  a  transition 
from  the  basic  record-oriented  approach  towards  a  model  supporting  more  human-oriented 
semantic  constructs.  While  traditional  data  models  store  data,  and  semantic  models  shift 
towards  a  more  human  knowledge  model,  [Til98]  suggests  a  third  method,  the  conceptual 
modeling  activity  geared  more  towards  the  domain-level  knowledge  and  program  under¬ 
standing  with  a  focus  on  the  end-user.  [Til98]  further  suggests  that  all  three  techniques 
may  be  more  suitable  than  one  method  if  there  are  different  categories  of  artifacts  (e.g., 
data,  knowledge,  information),  requiring  a  relational  model  physical  storage  of  data,  a  con¬ 
ceptual  model  for  representing  domain-level  knowledge,  or  a  semantic  model  for  interac¬ 
tive  discovery. 

[Wie93]  acknowledged  that  data  comes  from  many  diverse  and  heterogeneous 
sources  and  further  suggested  that  joining  heterogeneous  data  is  necessary  to  generate  in¬ 
formation.  Focusing  primarily  on  database  constructs  and  schema  research,  [WW90, 
Wie93,  WieOOa,  WieOOb,  WieOl]  developed  eight  viewpoints  for  differences  among  sys¬ 
tems,  and  [You02c]  added  a  ninth  viewpoint  of  heterogeneity  germane  to  this  research: 

•  Heterogeneity  of  Hardware  and  Software  systems, 

•  Heterogeneity  of  Organizational  Models 

•  Heterogeneity  in  Representation 

•  Heterogeneity  of  Scope, 

•  Heterogeneity  of  Level  of  Abstraction, 

•  Heterogeneity  of  Meaning, 

•  Heterogeneity  of  Temporal  Validity  [Wie93], 

•  Semantic  Heterogeneity  [WieOOa,  WieOOb], 

•  Heterogeneity  of  Structure  [Y ou02c] . 

This  research  revealed  progress  in  the  area  of  technical  interoperability,  including 
research  into  multilevel  simulation  of  discrete  network  models  [Ham96,  BKG+02],  distrib¬ 
uted  simulations  [Ham96,  HNP97],  and  the  Department-sponsored  development  of  the 
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HLA.  Although  there  has  been  progress  with  the  technical  interoperability  problem  since 
the  mid-1990s,  many  approaches  failed  to  improve  the  credibility  of  authoritative  represen¬ 
tations  and  reduce  the  federation’s  interoperability  problems  caused  by  substantive  interop¬ 
erability  anomalies.  Over  time  developers  modeled  many  of  the  same  objects  in  different 
ways,  potentially  causing  problems  when  they  interact  in  a  federation. 

[DSB+99,  HYOla,  HYOlb,  YPH01]  identified  systemic  substantive  interoperability 
issues  with  legacy  simulation  software  systems  and  the  inconsistent  representation  of  the 
same  entity  in  multiple  systems.  There  is  a  very  close  correlation  on  the  root  causes  be¬ 
tween  the  technical  and  substantive  interoperability  issues  cited  in  the  simulation  domain, 
and  the  previously  cited  work  of  [Wie93,  You02c]  on  information  from  heterogeneous 
sources.  However,  substantive  interoperability  remains  a  systemic  issue  persisting  through 
the  evolution  of  the  ALSP,  DIS  and  HLA  protocols,  and  remains  a  major  challenge  for 
achieving  truly  distributed  simulation  interoperability. 

[DB99]  defines  multiresolution  modeling  as  “building  a  single  model,  a  family  of 
models,  or  both  to  describe  the  same  phenomena  at  different  levels  of  resolution”  [DB99]. 
[DH92a]  noted  that  achieving  interoperability  in  diverse,  independently  developed  M&S 
with  different,  but  overlapping  resolutions,  based  on  different  assumptions,  limitations,  and 
perspectives  is  a  major  challenge  in  contemporary  M&S.  [Dav92a,  Dav93  and  DBM99]  in 
Figure  4-1  illustrate  that  the  tenn  resolution  represents  many  different  concepts  and  dimen¬ 
sions.  However,  [DavOO]  noted  MRM  may  not  be  sufficient  for  applications  requiring  dif¬ 
ferent  abstractions  or  perspectives,  that  vary  by  conception  of  the  system  or  use  of  vari¬ 
ables,  and  requires  a  new  model  construct  for  both  multiple  resolution  and  multiple  per¬ 
spectives  (MRMPM). 

In  [MDBOOa]  models  and  families  of  models  employed  hierarchical  trees  to  remain 
mutually  consistent  across  multiple  resolution  and  perspectives.  [BDMOO]  employed  an 
exploratory  analysis  technique  using  low-resolution  models  at  the  macro  level  of  analysis 
and  high-resolution  models  to  support  calibration  and  validation  efforts.  [TNM01]  later 
addressed  the  multi-resolution  question  in  federations,  and  identified  the  problems  that 
arise  from  federations  or  interaction  between  simulations  with  different  levels  of  detail  or 
abstraction  requiring  aggregation  and  disaggregation  of  information  required  by  an  HLA 
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federation.  [TNM01]  submit  two  new  approaches:  the  middleware,  and  the  federate-based 
approaches  to  replace  the  module-based  approach  to  implement  aggregation/disaggregation 
in  HLA  environments.  In  a  related  effort,  the  ARP  A/Tri-Services  sponsored  the  Rapid¬ 
prototyping  of  Application  Specific  Signal  Processors  (RASSP)  program  to  review  and 
compare  simulation  terminology,  interoperability  challenges,  and  previous  modeling  tax¬ 
onomies  as  a  basis  for  designing  a  multi-axis  taxonomy  to  describe  the  infonnation  content 
of  computer  model  types  and  define  abstraction  levels  supporting  development  of  interop¬ 
erable  models  [HMB+00]. 

Interoperability  poses  significant  challenges  in  many  areas  including  the  Command, 
Control,  Computers,  Communications,  Intelligence,  Surveillance,  and  Reconnaissance 
(C4ISR)  dimension.  [DoD02d]  defines  critical  Department  M&S  requirements  for  im¬ 
proved  C4ISR.  However,  there  are  currently  severe  M&S  to  C4ISR  shortcomings  and: 

•  Good  information  superiority  models  do  not  exist, 

•  Pre-scripted  battlefield  entities  are  non-responsive  and  lower  study  fidelity, 

•  Existing  models  are  simplistic,  speculative,  and  too  narrowly  focused,  lacking  the 
ability  to  do  end-to-end  analysis, 

•  Current  force-on-force  models  do  not  account  for  C4ISR  or  make  broad  assump¬ 
tions, 

•  Models  lack  of  system  performance  data, 

•  The  Department  needs  integrated  treatment  of  threat  signatures  and  sensor  mission 
management, 

•  Suitable  Opposing  Force  (OPFOR)  representations  are  lacking  [DoD02d]. 

A  SISO  M&S-to-C4I  Interoperability  Working  Group  report  [TFW+00]  identified 
that  uncoordinated  M&S  and  JTA  standards  have  been,  and  continue  to  be  developed  by 
both  communities,  and  several  factors  inhibit  a  completely  seamless  M&S-to-C4ISR  inter¬ 
face  including: 

•  Applicable  standards  (HFA,  JTA)  in  both  domains  continue  to  evolve  quickly, 

•  Interoperability  between  C4ISR  systems  or  components  may  span  multiple  levels, 

•  Each  domain  must  be  able  to  retain  authority  and  responsibility  for  data  V&V  and 
security, 

•  Each  domain  has  access  point  responsibilities  for  data  management,  metadata,  and 
conversions, 

•  Both  domains  must  support  active  interfaces  at  the  domain  boundaries, 

•  Both  architectures  require  a  translation  capability  until  future  architectures  employ 
the  same  data  elements  based  on  a  common  design  [TLW+00]. 
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An  M&S-to-C4ISR  interoperability  framework  proposed  by  [TLW+00]  includes  a  three- 
phase  plan  with  near-,  mid-,  and  long-tenn  solutions  evolving  from  a  limited  message- 
based  as-is  architecture.  The  projected  to-be  architecture  shares  common  databases,  im¬ 
plicit  interoperability,  “plug  and  play”  capabilities  and  full  duplex  information  exchange 
[TLW+00]. 
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V. 


THE  DEPARTMENT’S  FRAMEWORK  FOR  IMPROVED  CREDIBILITY 


A.  INTRODUCTION 

Chapter  V  provides  the  Department’s  current  architectural  framework  for  resolving 
the  underlying  systemic  causes  generating  the  pre-conditions  for  heterogeneous  system  rep¬ 
resentation  anomalies,  especially  in  federation  interoperability.  The  Department’s  initia¬ 
tives  to  improve  the  credibility  of  large-scale,  legacy  Department  M&S  supporting  distrib¬ 
uted  interoperability  include  the  “as-is”  architecture  and  the  evolving  “to-be”  architecture: 
the  Joint  Technical  Architecture,  and  the  Common  Technical  Framework.  The  Common 
Technical  Framework  components  include  the  High-Level  Architecture,  conceptual  model 
requirements,  and  data  standards  supporting  credible  simulation  development. 

B.  THE  DEPARTMENT’S  AS-IS  SIMULATION  ARCHITECTURE 

The  Department  established  tenns  of  reference  for  an  M&S  framework  to  improve 
the  communication  of  concepts,  interoperability,  and  development  of  authoritative  repre¬ 
sentations  including  architectural  implementations  (e.g.,  SIMNET,  ALSP,  DIS,  HLA).  The 
Common  Technical  Framework  (CTF)  for  M&S  contains  three  components:  the  HLA, 
CMMS,  and  data.  The  Federation  Development  and  Execution  Process  (FEDEP)  support 
the  implementation  of  the  HLA.  The  HLA  is  the  current  Department  and  IEEE  standard 
software  architecture  for  distributed  M&S  interoperability,  although  it  has  not  fully  re¬ 
solved  long-standing  technical  and  substantive  interoperability  issues.  Consequently,  rep¬ 
resentations  of  the  same  system  in  different  models  are  frequently  incompatible. 

Conceptual  Models  of  the  Mission  Space  (CMMS)  /  Functional  Description  of  the 
Mission  Space  (FDMS)  are  the  developer’s  method  for  developing  the  M&S  requirements 
into  a  detailed  designed  framework  to  develop  the  software.  The  verified  and  validated 
CMMS  /  FDMS  subsequently  supports  the  critical  role  of  verifying  the  software  develop¬ 
ment  effort.  Authoritative  data,  the  final  component  of  the  CTF  is  critical  to  the  Depart¬ 
ment’s  goal  of  improving  credibility  and  confidence  in  simulation  results,  however  authori¬ 
tative  data  may  be  the  single  most  pervasive  M&S  deficiency  area. 
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Software  architecture  theory  continues  to  evolve  in  academia,  the  private  sector, 
and  the  public  sector  including  the  Department.  Prior  to  the  development  of  the  HLA,  the 
Department’s  M&S  as-is  distributed  architecture  framework  consisted  of  the  Simulation 
Network  (SIMNET),  the  Aggregate  Level  Simulation  Protocol  (ALSP)126,  and  Distributed 
Interactive  Simulation  (DIS)  environments.  SIMNET  [MCU+95,  Gre99]  was  a  pioneering 
collaborative  effort  from  1983  until  1989  between  the  Defense  Advanced  Research  Projects 
Agency  (DARPA)  and  U.S.  Anny,  to  develop  distributed  simulations  operating  on  several 
interconnected  computers  and  determine  the  feasibility  for  distributed  simulations  to  pro¬ 
vide  a  training  capability  for  the  U.S.  Army. 

The  ALSP  distributed  network  protocols  patterned  after  the  SIMNET  effort 
[WSW91,  WWG93]  permits  multiple,  pre-existing,  large-scale  aggregated-level,  construc¬ 
tive  warfare  simulations  to  interact  in  logical  time  over  local  or  wide  area  networks,  control 
its  own  objects  and  share  information  with  other  simulations.  ALSP,  originally  developed 
by  DARPA  in  1992,  and  currently  managed  by  the  U.S.  Army  Simulation,  Training  and 
Instrumentation  Command  (STRICOM)  through  a  multi-service  agreement,  was  based  on 
four  principles,  ~  and  a  distributed  architecture  composed  of  a  three-part,  two-layer  proto- 
col  component  and  two  software  components,  "  permitting  interoperation  of  dissimilar 
Service  and  Joint  constructive  M&S  [WSW91,  WWG93]. 

Early  ALSP  confederations  of  Service  simulations  (e.g.,  Corps  Battle  Simulation, 
Air  Warfare  Simulation)  supported  several  joint  and  combined  training  exercises  (e.g.,  At¬ 
lantic  Resolve,  Unified  Endeavor,  Ulchi  Focus  Lens)  [DoD95],  The  Joint  Training  Con¬ 
federation,  initially  deployed  for  major  training  exercises  in  1992,  evolved  into  the  largest 
ALSP  confederation  with  eight  interacting  simulations  [Log02].  The  ALSP  confedera¬ 
tion  simulations,  developed  originally  as  stand-alone  systems,  support  only  limited  interop¬ 
erability,  require  lengthy  set-up  times,  lack  interfaces  to  real-world  systems,  and  need  large 

126  ALSP  is  a  family  of  M&S  interface  protocols  and  supporting  infrastructure  software  that  permits  the  integration  of  distinct  M&S  and 
war  games.  Combined,  the  interaction  protocols  and  software  enable  large-scale,  distributed  M&S  and  war  games  of  different  domains 
to  interact  at  the  combat  object  and  event  level  [DoD98]. 

127 

ALSP  principles:  1)  Distributed  computation  based  on  combat  entity  ownership,  2)  avoidance  of  single  critical  resources,  3)  reliance 
on  broadcast  communications,  and  4)  replication  of  a  limited  set  of  combat  entity  attributes  among  all  M&S  [WSW91]. 

The  ALSP  protocol  component  exists  of  1)  a  peer  level  protocol  joins  translators  and  object  attribute  management,  2)  a  connection 
protocol  manages  the  translator  to  gateway,  and  3)  a  second  peer  level  protocol  that  joins  gateways  and  deals  with  timing  issues.  The 
ALSP  software  components  include  the  translators  and  the  gateways  [WSW91]. 

129  ALSP  interacting  simulations:  AWSIM,  CBS,  RESA,  MTWS,  CSSTSS,  JQUAD,  TACSIM,  and  PSM  [Log02], 
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amounts  of  manpower.  The  Department  plans  to  replace  ALSP  confederations  with  the 
Joint  Simulation  System. 

The  Distributed  Interactive  Simulation  (DIS)  protocol  [MCU+95,  IEE98a]  provides 
a  synthetic  interactive  networked  environment  in  which  humans  may  interact  real  time, 
through  the  connection  of  different  platform-level  virtual  M&S,  simulators,  or  instru¬ 
mented  live  exercises.  The  DIS  protocols  and  standards  [IEE98a]  establish  a  common  data 
exchange  environment  or  common  messaging  environment,  using  Protocol  Data  Units, 
supporting  the  interoperability  of  heterogeneous,  geographically-distributed  live,  virtual, 
and  constructive  simulations  [DoD95].  The  DIS  synthetic  environment  uses  verified  sce¬ 
narios,  tactics,  techniques,  and  procedures  to  train  testers  on  new  hardware  or  software 
[AWQ+93],  and  conduct  trial  test  runs  before  executing  costly  field  tests. 

The  DIS  protocol  development  effort  also  experienced  broad  participation  from  an 
open  forum  of  industry,  government,  and  academia,  and  plays  a  central  role  in  the  Depart¬ 
ment’s  M&S  portfolio.  [LBV96,  Gre99]  cited  key  DIS  techniques  including  a  “dead  reck¬ 
oning”  methodology  based  on  a  world  and  body  coordinate  system  to  minimize  communi¬ 
cation  requirements  between  simulators  in  order  to  produce  a  realistic  and  credible  virtual 
battlefield.  However,  the  computation-intensive  DIS  protocol  requires  high  bandwidth  lev¬ 
els  for  it’s  broadcast  messaging  technique,  and  lacks  support  for  different  time  management 
methods,  realistic  command  and  control  representations,  and  dynamic  changes  in  the  envi¬ 
ronment  [DoD95]. 

C.  THE  DEPARTMENT’S  TO-BE  ARCHITECTURE 

The  Department’s  future  “to-be”  architecture  development  includes  the  evolving 
Joint  Technical  Architecture  (JTA),  common  Operating  Environment  (COE)  and  C4ISR 
Framework  supported  by  the  Technical  Reference  Model  (TRM)  with  a  major  focus  to  im¬ 
prove  interoperability  and  information  exchange.  The  Department’s  COE,  JTA,  and 
C4ISR  framework  support  the  foundation  of  the  software-intensive  “to-be”  architecture. 

The  Department’s  “to-be”  architecture  or  target  architecture  will  evolve  to  meet  the  per¬ 
formance  requirements  codified  by  the  Government  Performance  and  Results  Act  of  1993 
and  Clinger-Cohen  Act  of  1996  [CC96],  the  legal  frameworks  for  developing  United  States 
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Government  enterprise-level  information  technology  architecture  standards  and  policy 
[LSF+01].  The  Department’s  “to-be”-architecture  will  also  reflect  the  emerging  Federal 
Enterprise  Architecture  standards  identified  by  [LLT+99]  and  approved  by  the  Chief  In¬ 
formation  Officer  (CIO)  council  [HamOOa]  under  the  direction  of  Executive  Order  13011, 
for  Federal  Information  Technology. 

The  Department’s  Enterprise  Integration  (El)  Implementing  Strategy  and  Enterprise 
Model  described  by  [DoDOOc]  identifies  near-  and  long-term  strategies  including  the  exist¬ 
ing  Corporate  Information  Management  strategy,  an  integrated  technical  architecture 
framework  for  infonnation  management,  data  standards,  and  shared  databases.  The  long¬ 
term  El  strategy  provided  by  [DoDOOc]  “ensures  consistency,  quality,  timeliness,  availabil¬ 
ity,  and  security  of  shared,  corporate  data  by  implementing  corporate  databases  using  stan¬ 
dard  data  elements  as  soon  as  possible”  [DoDOla], 

The  Defense  Information  Infrastructure  ( DII)  Common  Operating  Environment 
(COE)  [Har97,  Har98a,  DoDOOd]  emerged  in  the  mid-1990s  as  a  foundation  for  developing 
open  systems  with  a  “plug  and  play”  open-architecture  capability  based  on  the  open  techni¬ 
cal  architecture  developed  for  the  Global  Command  and  Control  System  [HPS98].  The  DII 
COE,  an  architectural  approach  for  building  interoperable  systems,  includes  a  collection  of 
reusable  components,  a  software  infrastructure,  and  a  set  of  standards  and  guidelines  for  an 
open  architecture  designed  around  a  client  /  server  model  [BPM+95].  The  DII  COE  also 
contains  products  to  meet  operational  requirements,  which  are  not  fully  JTA  compliant 
[HB00,  JTAOla];  however,  the  goal  is  to  evolve  to  full  compliance  with  the  appropriate 
JTA  standard.  In  support  of  this  goal,  [HSOOa,  CHOI,  CM03]  presented  M&S  infrastruc¬ 
ture  alternatives  to  better  align  the  current  M&S  architecture  with  the  DII  COE. 

The  Joint  Technical  Architecture  (JTA)  [JTA02a,  JTA02b]  and  the  C4ISR  Architec¬ 
ture  Framework  [AWG97]  provides  the  Department  with  the  basis  for  developing  systems 
with  the  required  seamless  interoperability,  defining  service  areas,  interfaces,  tenninology, 
and  standards  applicable  to  all  systems  and  mandated  for  the  management  and  development 
of  new  or  improved  Department  systems  [Ste02a],  The  Department  implemented  the  JTA 
by  defining  an  interrelated  set  of  operational,  technical,  and  system  views.  The  JTA  also 
builds  on  service  areas  identified  in  the  Technical  Reference  Model  [BKOla,  TRMOla, 
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TRMOlb],  supported  by  technically  stable,  mature  commercial  standards  and  guidelines. 
The  JTA  [JTA02a]  consists  of  two  main  parts:  the  JTA  Core  and  the  JTA  Domains  with 
sub-domains  shown  in  Figure  5-1,  the  JTA  Hierarchy  Model. 


Figure  5- 1 .  JTA  Hierarchy  Model  (From  [JTA02a]) 


The  JTA  [JTA02a]  core  information  technology  categories  include  information 
processing  standards,  information  transfer  standards,  infonnation  modeling,  metadata,  in¬ 
formation  exchange  standards,  human-computer  interface,  and  information  security  stan¬ 
dards.  The  JTA  also  identifies  the  minimum  set  of  JTA  elements  applicable  to  all  Depart¬ 
ment  systems  to  achieve  interoperability  and  domain-specific  elements  to  ensure  interop¬ 
erability  within  the  domain  (e.g.,  C4ISR,  Combat  Support,  Modeling  and  Simulation,  and 
Weapon  Systems),  but  not  necessarily  for  inter-domain  interoperability  between  the  four 
JTA  domains  or  their  subordinate  domain  elements  [JTA02a]. 

The  Department’s  Technical  Reference  Model  (TRM)  [TRMOla,  GMWOOb]  and 
TRM  User  Guide  [TRMOlb]  support  the  requirements  of  increasingly  complex  and  diverse 
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systems  by  integrating  the  service  and  interface  views,  and  identifying  interfaces  and  con¬ 
tent.  The  [TRMOla]  evolved  from  the  Department’s  Technical  Architecture  Framework 
for  Information  Management  (TAFIM),  and  includes  three  major  model  elements  (1)  the 
services,  (2)  the  interfaces  and  (3)  the  entities.  The  three  major  model  elements  specified 
by  the  [TRMOla]  include  the  following  characteristics:  the  ability  to  support  system  archi¬ 
tectures  [Dem95b,  MR02],  model  degree  of  freedom  to  select  or  expand  on  services  and 
interfaces,  and  the  ability  to  support  new  services. 

The  [TRMOla]  also  provides  interface  definitions,  and  environment  configurations, 
supports  a  model’s  ability  for  different  views,  and  provides  methods  of  mapping  the  model 
to  other  known  reference  models.  In  a  related  effort  within  the  M&S  domain,  the  SISO 
C4ISR/  Simulation  Technical  Reference  Model  study  group  [BBM+01,  GLT+02]  devel¬ 
oped  a  standard  frame  of  reference  for  interoperability  between  C4ISR  systems  and  M&S 
systems.  [RHS99]  also  developed  a  supporting  conceptual  model  with  an  infonnation  ex¬ 
change  model  focused  on  the  information  exchange  between  C4ISR  and  M&S  components. 

The  Department  developed  the  JTA  [JTA02a,  JTA02b]  and  the  C4ISR  Architecture 
Framework  [AWG97]  to  improve  the  system  architecture.  However,  [CBB+03]  caution 
that  the  C4ISR  Architecture  Framework  [AWG97]  almost  exclusively  documents  the  sys¬ 
tem  architecture,  and  that  none  of  the  three  [AWG97]  views  (e.g.,  operational,  systems, 
technical),  or  the  essential  or  supporting  products,  “prescribe  anything  that  remotely  re¬ 
sembles  software  architecture”  [CBB+03],  This  assessment  of  the  maturity  of  the  Depart¬ 
ment’s  software  architecture  including  [AWG97]  is  a  systemic  issue,  since  [IEEOOb]  notes 
there  is  no  single,  accepted  framework  codifying  software  architectures,  despite  significant 
research  in  the  area. 

The  Modeling  and  Simulation  Domain  prescribed  by  [TRMOla]  includes  the  De¬ 
partment’s  Common  Technical  Framework  (CTF)  [DoD95,  GMS+96]  for  facilitating  M&S 
interoperability  and  reuse.  The  CTF  has  three  components:  the  High-Level  Architecture 
(HLA)  to  which  the  Department’s  which  M&S  must  conform;  Conceptual  Model  of  the 
Mission  Space  (CMMS)  to  provide  a  basis  for  the  development  of  consistent  and  authorita¬ 
tive  representations;  and  data  standards  to  provide  common  representations  of  data  across 
M&S  and  C4I  systems  [DoD95,  GMS+96].  The  Department’s  M&S  architecture  efforts 
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cited  by  [DoD95,  GMS+96]  focused  on  interoperability  and  improving  the  shortfalls  of  the 
previous  interoperability  protocols  (e.g.,  ALSP,  DIS). 

1.  High-Level  Architecture  (HLA) 

The  [DoD95]  defined  the  requirement  for  a  common  high-level  simulation  architec¬ 
ture  to  facilitate  interoperability  of  all  types  of  simulations,  interoperability  with  C4I  sys¬ 
tems,  and  improved  reuse  of  M&S  components.  In  the  Department’s  M&S  domain,  the 
current  standard  software  architecture  for  distributed  simulation  interoperability  is  the 
HLA.  Under  development  since  1995  and  designated  the  technical  architecture  for  all  De¬ 
partment  M&S  [GanOO],  the  HLA  technical  architecture  includes  three  major  components: 
the  simulation  member  of  the  federations,  or  federate,  the  runtime  infrastructure  (RTI),  and 
the  runtime  interface.  The  HLA,  is  also  an  IEEE  standard  [IEEOOc]  with  a  reusable  com¬ 
mon  distributed  framework,  composed  of  an  HLA  rule  set  [IEE98e,  IEEOOc],  the  HLA  in¬ 
terface  specification  [IEEOOd,]  and  the  HLA  Object  Model  Template  (OMT)  [IEEOOe, 
LutOO],  supporting  a  wide  range  of  M&S  application  areas  at  different  resolutions. 

The  HLA  rules  [IEEOOc]  are  divided  into  two  major  categories:  federate  and  federa¬ 
tion  rules.  An  HLA  federate  may  be  another  computer-based  simulation,  a  manned  simula¬ 
tor,  an  interface  to  a  live  or  instrumented  range,  or  a  simulation  utility.  An  HLA  federation 
is  a  set  of  interacting  simulations  or  federates  represented  by  a  federation  object  model 
(FOM)  based  on  the  OMT  format  [IEEOOc,  BRA02].  The  HLA  OMT  [IEEOOe]  specifies 
the  object  model  and  the  essential  objects,  attributes,  and  interactions,  shared  across  the 
federation,  supporting  interoperability.  An  HLA  federate  requires  a  simulation  object 
model  (SOM)  based  on  the  OMT,  to  identify  all  public  information  shared  across  the  fed¬ 
eration  [IEEOOc]. 

A  FOM  identifies  the  essential  classes  of  objects,  object  attributes,  and  object  inter¬ 
actions  [IEEOOe],  supported  by  the  HLA  federation.  At  runtime  in  an  HLA  federation,  1) 
the  federate  manages  all  object  representations,  and  2)  only  one  federate  owns  any  speci¬ 
fied  attribute  of  an  instance  of  an  object  at  one  time.  The  HLA  interface  specification  de¬ 
scribes  the  run  time  services  provided  by  the  RTI  to  a  federate,  and  prescribes  the  interface 
from  the  federate  to  the  RTI,  with  six  classes  of  management  services:  federation,  declara- 
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tion,  object,  ownership,  time  (continuous,  time-stepped,  discrete  event),  and  data  distribu¬ 
tion  [IEEOOd]. 

In  addition,  the  HLA  supports  the  passive  collection  of  data  and  the  interface  to  live 
participants  using  the  HLA  interface  to  interact  with  the  RTI  [Dah99].  The  HLA  runtime 
interface  specification  supports  the  following  RTI  activities:  1)  a  standard  for  federation 
interaction  with  the  RTI,  2)  the  method  to  invoke  RTI  services  supporting  federation  run¬ 
time  interactions,  and  3)  the  ability  of  a  federate  to  interact  with  the  RTI  [IEEOOd].  Several 
HLA  interoperability  capabilities  and  lessons-learned  identified  by  [Tol02]  have  been  ad¬ 
dressed  in  federation  calibration  experiments  [BLR+02],  federating  time-stepped  and  dis¬ 
crete  event  simulations  [Bee02],  and  object-oriented  architectures  [CA02],  [Dah99, 
HarOlc]  described  the  current  technical  status  of  the  HLA  and  identified  future  challenges. 
Other  initiatives  include  component-based  extensions  [RPK02];  employing  the  Extensible 
Markup  Language  (XML)  [SBOO]  and  Unified  Modeling  Language  (UML)  methods  sug¬ 
gested  by  [BRJ99,  OMG99,  SB01,  Oes02,  and  MB02]  to  improve  HLA  documentation; 
and  distributed  test  and  evaluation  implementations  [Har98b]. 

The  HLA  RTI  according  to  [Dah99,  CA02]  acts  as  the  distributed  operating  system 
providing  the  interface  specification  and  management  services  to  the  federation.  [BJ98] 
compared  the  HLA  with  other  private  sector  distributing  computing  efforts  including  the 
Common  Object  Request  Broker  (CORBA)  sponsored  by  the  Object  Management  Group 
(OMG),  and  the  Remote  Method  Invocation  (RMI)  from  SunSoft’s  Java  Development  Kit 
(JDK)  [WWN97],  and  identified  the  strengths  and  weaknesses  of  each  approach.  [BCB02] 
discussed  methods  to  manage,  monitor,  and  manage  federations.  Viewing  federation  per¬ 
formance  as  a  critical  factor,  [FMF02]  explain  RTI  benchmark  studies  comparing  three  RTI 
implementation  approaches,  while  [HW02]  described  the  possible  implementation  of  data 
distribution  management  services  in  the  next-generation  RTI. 

Community  researchers  continually  provide  recommendations  for  HLA  improve¬ 
ments  and  enhancements.  [HHS+98]  suggests  specifying  additional  optional  classes  of  in¬ 
formation  allowing  the  development  of  a  more  complete  description  of  the  federation  struc¬ 
ture  or  behavior.  [HHS+98]  also  explains  the  capstone  definition,  class  terminology,  and 
classification  rules  of  a  reference  FOM.  [RDOO]  describe  several  additional  FOMs,  devel- 
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oped  according  to  the  format  described  in  the  HLA  OMT,  including  a  prototype  C4I  FOM. 
[GRH+01,  GHL+01,  and  GHO+Ol]  describe  and  specify  the  Base  Object  Model  (BOM);  a 
SISO-approved  type  of  reference  FOM  designed  to  provide  a  component  overlay  capability 
to  the  HLA  architecture.  Other  FOM  initiatives  include  an  intelligence  and  electronic  war¬ 
fare  FOM  developed  by  [WS02]  to  bridge  constructive,  virtual  and  live  systems  in  a  syn¬ 
thetic  environment  to  improve  interoperability;  a  federation  object  model  for  atmospheric 
dispersion  proposed  by  [HCP+02];  and  a  real-time  platform  reference  FOM  suggested  by 
[OF02], 

In  order  for  the  Department  to  achieve  M&S  interoperability  goals,  [MCL+99] 
identified  several  issues  including  the  need  for  “a  better  means  of  agreeing  and  translating 
alternative  model  data  representations  (e.g.,  model  interoperability)”  [MCL+99].  In  sup¬ 
port  of  improved  simulation  interoperability,  [LLP+02]  conducted  and  analyzed  calibration 
exercises  to  baseline  performance  data,  while  [CCB+02]  described  the  development  of  an 
HLA-compliant  high-perfonnance  computing  RTI  to  improve  RTI  interoperability  per¬ 
formance  issues  identified  by  [MCL+99,  FMF02],  The  FOM  and  OMT  (and  their  relation¬ 
ship  with  the  RTI)  also  need  extensions  and  better  definitions  according  to  [MCL+99]. 

After  initial  HLA  federation  experimentation  and  prototype  efforts  experienced  a 
great  deal  of  trial  and  error,  the  Department  M&S  community  detennined  the  need  for  ad¬ 
ditional  guidance  to  improve  cross-domain  interoperability  and  future  federation  collabora¬ 
tion.  The  community  achieved  consensus  for  various  “best  practices”  aspects  of  federation 
development,  leading  to  the  establishment  of  a  common  process  view  for  HLA  federations 
addressing  the  spectrum  of  interests  within  the  HLA  community,  and  introduced  the  Fed¬ 
eration  Development  and  Execution  Process  (FEDEP)  [DMS99]  in  September  1996,  fol¬ 
lowed  by  a  release  of  the  FEDEP  concept  of  operations  in  1997. 

The  FEDEP  six-step  process  illustrated  in  Figure  5-2  continues  to  evolve  as  the 
community  experience  identifies  improvements  [Wai97,  Wai02a,  DMS99]  and  efforts  are 
currently  underway  to  propose  FEDEP  IEEE  guidance  for  future  HLA  federation  construc¬ 
tion  methods  [LDG+02].  In  a  continuing  trend  to  perform  only  inherently  governmental 
functions,  the  Department  transitioned  the  RTI  development  responsibilities  from  a 
DMSO-sponsored,  government-funded  project  to  the  private  sector  for  future  development 
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and  capitalization  in  October  2002.  In  December  2003,  researchers  at  the  Johns  Hopkins 
Applied  Physics  Laboratory  (APL)  reported  the  first  successful  commercial  application  of 
the  IEEE  1616  HLA  specification  in  a  medical  federation  between  the  Massachusetts  Insti¬ 
tute  of  Technology  cardiovascular  system  simulation  and  the  APL’s  detailed  model  of  both 
the  left  and  right  ventricles  [JHU03]. 


PROGRAM  AVAILABLE 

OBJECTIVES  RESOURCES 


Figure  5-2.  The  FEDEP  Process  (From  [DMS99]) 

2.  Conceptual  Model  of  the  Mission  Space  (CMMS) 

A  conceptual  model  [DoD95,  GMS+96]  is  the  developer’s  method  of  translating  the 
M&S  requirements  into  a  detailed  design  framework  to  develop  the  software,  and  describes 
the  conceptual  model  components,  interactions,  and  the  M&S  concept  of  operations.  Sev¬ 
eral  researchers,  including  [SPC+98,  PacOOa,  SHOO,  SheOOb,  DDH01]  validated  the  basic 
need  for  conceptual  models,  and  developed  supporting  perspectives  of  conceptual  modeling 
theory,  identified  systemic  issues,  and  proposed  solutions.  However,  the  Department’s 
management  of  conceptual  models  has  two  major  challenges  to  overcome,  an  inconsistent 
past  and  a  demanding  future. 

Today,  the  Department’s  M&S  community  currently  lacks  consensus  and  standards 
for  conceptual  models.  Moreover,  conceptual  models  are  critical  to  the  key  Department’s 
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M&S  issues  of  authoritative  representations,  improved  simulation  credibility,  enhanced 
confidence  in  simulation  results,  integrated  simulation  security,  and  improved  simulation 
interoperability.  Surveying  the  current  M&S  situation,  [LRH+01]  consider  the  term  “con¬ 
ceptual  model”  extremely  overloaded  and  overused  within  the  M&S  community.  For  ex¬ 
ample,  the  tenn  “conceptual  model”  describes  the  first  abstraction  of  representations  in  an 
M&S  and  describes  a  high-level  design  of  how  all  the  components  in  an  M&S  relate  to  an¬ 
other.  The  wide  and  varied  body  of  literature  on  the  subject  of  conceptual  models  is  an  in¬ 
dication  of  their  importance,  but  also  reveals  a  lack  of  consensus  on  conceptual  model 
methods,  verification  and  validation,  fonnat  and  development  methodology. 

A  short  survey  on  the  breadth  of  conceptual  modeling  research,  concepts,  theories, 
and  the  different  perspectives  include  the  following  widely-diverse  conceptual  model  top¬ 
ics:  verification  and  validation  [Tho97b,  Wai97,  Pac98a,  Pac98c,  SPC+98,  Pac99a, 
PacOOa,  PacOOb,  PacOOc,  MetOO,  LRH+01,  PacOla,  Bor02],  conceptual  model  object- 
oriented  schemas  [Wai97],  conceptual  model  quality  [TB97],  and  conceptual  model  docu¬ 
mentation  [Tho97b,  Pac99a].  The  literature  also  includes  significant  information  on:  con¬ 
ceptual  model  environment  [Bir98],  conceptual  model  aggregation  /  disaggregation 
[BidOO],  fidelity  [Pac98c,  PacOOa,  PacOOc,  PacOla],  conceptual  models  and  the  mission 
space  [MM98],  conceptual  model  validation  [Pac98c],  conceptual  model/  functional  de¬ 
scription  of  the  mission  space  transition  [HM98,  JorOl],  multiple  conceptual  model  ap¬ 
proaches  [Whi99],  and  conceptual  model  reuse  [PacOOc,  Pac02a].  In  addition  the  literature 
search  revealed  addition  citations  for  aircraft  conceptual  models  [ChaOO],  assumptions 
document  [LKOO],  joint  conceptual  models  [Bir98,  MetOO,  LRH+01],  and  conceptual  mod¬ 
els  for  HLA  federations  [Tho97b,  LC97,  Wai97,  Pac98c,  Bir98,  PacOla]. 

The  conceptual  model  may  also  include  the  algorithms,  equations  as  well  as  any  as¬ 
sumptions  and  limitations.  However,  even  the  naming  convention  for  the  Department’s 
conceptual  models  is  challenging  with  the  CMMS  [DoD95,  GMS+96]  term  and  the  newer 
Functional  Description  of  the  Mission  Space  (FDMS)  [RPGOO]  tenn  used  interchangeably. 
In  this  effort  we  will  nonnally  use  the  use  the  tenn  conceptual  model  unless  specifically 
addressing  the  CMMS-  or  FDMS-versions  of  the  conceptual  model  methodology. 
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However,  whichever  term  or  format  is  used,  whether  its  is  the  older  CMMS  term  or 
the  newer  FDMS  label,  [GMS+96]  cites  a  Department’s  key  concern  with  M&S  credibility 
as  the  lack  of  a  detailed  and  documented  conceptual  model  and  design  specification  for  ex¬ 
isting  legacy  systems  M&S.  Conceptual  model  verification  is  the  Department’s  process  for 
ensuring  the  conceptual  model  meets  specified  requirements.  However,  the  conceptual 
model  documentation  for  many  legacy  M&S  may  be  inadequate,  if  it  exits  at  all,  according 
to  [Pac99b].  The  conceptual  model  for  simulations  suggested  by  [PacOOa]  addresses  the 
simulation’s  requirements,  context,  entities  and  processes. 

[Had98]  proposed  a  specification  of  metrics  for  describing  existing  CMMS  and  for 
guiding  subject  matter  experts  in  the  acquisition  and  documentation  of  the  Military  Opera¬ 
tions  Mission  Space.  [FY97]  decomposes  his  conceptual  model  design  and  M&S  imple¬ 
mentation  according  into  a  layered  structure  to  account  for  model  dependency,  information 
flow,  and  model  control;  and  envisions  three  different  abstractions:  the  real  world,  the  con¬ 
ceptual  model  of  the  mission  space  and  the  software  implementation.  Additional  perspec¬ 
tives  on  the  definition  and  purpose  of  conceptual  models,  where  conceptual  models  fit  into 
the  M&S  development  lifecycle,  and  how  conceptual  models  are  developed  include  con¬ 
cepts  forwarded  by  [GMS+96,  Hom+97,  SPC+98,  DMS98a,  DMS98b,  DMSOOa,  PacOOa, 
RPGOO,  Bor02,  Pac02d].  A  properly  developed  conceptual  model  for  the  M&S  is  the  best 
generic  validity  referent  according  to  [PacOlb],  since  it  captures  both  pertinent  elements  of 
system  theory  and  intended  M&S  application. 

Today,  there  still  exist  several  systemic  impediments  to  the  development  of  credible 
explicit  conceptual  models.  Two  major  issues  addressed  by  [Pac02a]  are  the  lack  of  formal 
requirements  or  standards  for  conceptual  models  and  the  lack  of  an  explicit  M&S  concep¬ 
tual  model  as  a  defined  contract  deliverable.  There  are  also  issues  identified  by  [PacOOa] 
with  abstracting  representation  from  available  information,  or  for  describing  and  document¬ 
ing  the  conceptual  model.  In  addition,  [Bor02]  is  concerned  about  the  level  of  understand¬ 
ing  and  development  of  conceptual  models  of  the  mission. 

Without  a  verified  and  validated  conceptual  model  (e.g.,  CMMS  or  FDMS)  upon 
which  to  base  the  verification  of  the  software  implementation,  and  ultimate  validation  of 
the  simulation,  it  may  be  extremely  difficult,  if  not  technically  impossible  to  credibly  fol- 
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low  the  Department’s  recommended  guidelines  for  verification  and  validation  [GMS+96, 
RPGOO],  This  may  result  in  an  adverse  opinion  of  the  simulation’s  credibility,  which  un¬ 
dermines  user  confidence  in  the  simulations  results. 

Definitions  of  conceptual  model  validity,  model  verification,  operational  validity, 
data  validity,  and  recommended  procedures  forwarded  by  [Sar98]  include  the  relationship 
of  V&V  to  the  development  process.  Models  based  on  repeatable,  measurable  phenomena 
are  usually  empirical  according  to  [Mey98]  while  simulations  are  normally  probabilistic  in 
nature  since  the  supporting  phenomena  may  be  unknown,  uncertain  or  questionable. 
[Mey98]  also  proposed  the  possible  valid  uses  of  a  simulation  if  the  real-world  phenomena 
are  unavailable,  while  acknowledging  the  possible  invalid  use  of  a  model  without  knowing 
the  supporting  phenomena.  [PacOla]  also  provides  ideas  on  how  to  perform  validation  as¬ 
sessments  when  there  is  inadequate  information  about  one  or  more  federate  conceptual 
models. 

[Had98]  describes  the  problem  of  defining  an  appropriate  level  of  resolution  for  the 
CMMS,  and  addresses  ways  that  system  purpose  and  desired  capabilities  can  drive  the  de¬ 
velopment  of  prescribed  requirements  for  CMMS.  In  a  follow-on  SISO-sponsored  effort, 
[Gro99  and  RGHOO]  proposed  a  new  CMMS  methodology  to  define  the  context  for  the  de¬ 
velopment  and  support  of  simulation  object  models  and  federation  object  models,  based  on 
describing  and  developing  several  CMMS  corresponding  to  the  battlespace  of  real  mission 
areas,  including  sufficient  infonnation  to  support  the  new  CMMS  role  as  a  referent  for  fi¬ 
delity  measures.  The  overarching  CMMS  framework  envisioned  by  [Gro99  and  RGHOO] 
would  provide  access  to  the  necessary  detailed  data  and  support  the  development  of  consis¬ 
tent  and  interoperable  authoritative  representations  within  a  standard  fidelity  framework. 
More  importantly  an  expanded  CMMS  model  provides  a  possible  solution  to  the  systemic 
issue  of  how  to  define  a  standard  fidelity  referent,  which  has  been  a  major  obstacle  to  the 
development  of  accepted  definitions  of  fidelity,  which  [Gro99  and  RGHOO]  contend  is 
needed  for  supporting  fidelity-based  verification  and  validation  process. 

Conceptual  models  also  support  the  knowledge  acquisition  process  and  provide  a 
common  starting  point  for  constructing  consistent  and  authoritative  M&S  representations. 
The  knowledge  acquisition  alternatives  addressed  by  [TS92,  CPS+96,  Ede97,  Mai97, 
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DRC99,  KB099,  DSB+01]  support  the  development  of  a  valid  conceptual  model.  A  con¬ 
ceptual  model  framework  endorsed  by  [PacOOa]  identifies  four  steps  in  developing  a  con¬ 
ceptual  model,  and  proposes  greater  use  of  fonnal  methods  to  develop  M&S  requirements 
and  define  the  conceptual  model.  [HOM+97,  RPGOO]  specify  a  CMMS  technical  frame¬ 
work  providing: 

•  Technical  standards, 

•  Administrative  procedures, 

•  Operational  infrastructure 

•  The  common  semantics  and  syntax  for  describing  the  mission  space, 

•  A  common  format  database  management  system  [RPGOO], 

•  A  closed-loop  engineering  process  for  creating  and  maintaining  conceptual  models, 

•  Data  interchange  format  (DIF)  standards  for  conceptual  model  integration  and 
interoperability  [HOM+97,  RPGOO]. 

Several  DMSO  sponsored  reports  [HSB98,  SheOOb,  HSOOa,  HSOOb]  address  the 
state  of  current  conceptual  model  development  methods  and  tools  developed  by  and  for  the 
Department’s  M&S  community.  A  recent  comprehensive  assessment  of  the  DMSO 
CMMS  support  effort  by  [LuqOl]  addressed  the  following  areas:  a)  the  effectiveness  of 
software  risk  assessment  models,  b)  enhancements  to  the  FDMS  resource  center,  c)  XML 
and  wrapper-based  translators  for  Department  databases,  and  d)  metrics  for  selecting  auto¬ 
mated  test  tools;  and  suggested  improvements  based  on  the  latest  research. 

Most  experts  agree  on  the  need  for  conceptual  models,  although  a  wide  variety  of 
interpretations  and  a  great  deal  of  confusion  for  conceptual  modeling  exists  within  the  De¬ 
partment’s  M&S  community  today;  with  consensus  difficult  to  achieve  on  the  exact  defini¬ 
tion  of  conceptual  modes  [LRH+01].  As  a  result,  the  current  draft  [DoDOla]  cites  a  need 
for  developing  new  and  improved  V&V  technologies  and  methodologies,  and  improve¬ 
ments  for  the  V&V  of  representations. 


130  Every  language  of  communication  possesses  two  identifiable  properties,  the  form  of  the  language  and  the  meaning  associated  with 
the  form.  The  syntax  of  the  language  is  a  set  of  rules  specifying  which  forms  of  the  language  are  grammatically  acceptable,  its  grammar. 
The  meaning  derived  from  a  syntactically  correct  instance  of  a  language  may  be  viewed  from  two  points,  the  meaning  intended  by  the 
originator,  the  semantics  of  the  language,  or  the  meaning  interpreted  by  a  receiver,  the  pragmatic  meaning  [RR83]. 
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3. 


Functional  Description  of  the  Mission  Space  (FDMS) 


The  term  Functional  Description  of  the  Mission  Space  (FDMS)  [RPGOO,  DouOl] 
replaced  the  previous  Conceptual  Model  of  the  Mission  Space  (CMMS)  term,  in  the  De¬ 
partment’s  M&S  lexicon,  as  a  required  component  of  the  M&S  development  and  V&V 
processes.  The  FDMS  provides  the  real-world  description  of  entities,  processes,  and  activi¬ 
ties  for  the  design  phase,  and  detailed  representations  of  the  problem  domain  in  the  re¬ 
quirements  development  phase  of  the  simulation  life  cycle.  Most  of  the  conceptual  models 
[LRH+01]  described  support  classification  either  as  a  domain-oriented  classification,  such 
as  the  battlespace  [PRM97],  or  design-oriented  classification.  There  are  four  basic  formats 
to  describing  conceptual  models  identified  by  [RPGOO]:  the  ad  hoc  method,  design  ac¬ 
commodation,  development  paradigm,  and  the  scientific  paper  approach.  The  FDMS  em¬ 
ploys  the  development  paradigm  method,  although  the  [RPGOO]  recommends  the  scientific 
paper.  The  ad  hoc  approach  is  the  most  common  fonnat  employed  today  to  develop  the 
conceptual  model  [RPGOO].  The  M&S  developer  normally  produces  the  FDMS. 

4.  Data  Standards 

Data  standards  are  the  third  component  of  the  common  technical  framework  for 
M&S.  In  any  M&S  application,  the  associated  data  quality  should  be  as  creditable  as  the 
M&S  itself  if  the  user  or  sponsor  is  to  attach  confidence  to  the  results.  [DoD95]  defines 
data  quality  for  the  Department’s  M&S  programs.  A  proposed  draft  [DoDOla]  places 
increased  emphasis  on  data  quality  adding,  “That  a  model  implementation  and  its  associ¬ 
ated  data  accurately  represents  the  developer’s  conceptual  description  and  specification” 
DoDOla].  [BCE+99]  also  suggests  data  quality  attributes  including  source  accuracy,  fidel¬ 
ity,  suitability,  credibility,  maturity,  cost,  availability,  and  similar  characteristics.  [Kil02] 
recommends  data  accuracy  and  quality  information  supported  by  a  review  at  three  levels: 
database,  data  element,  and  data  value;  employing  descriptive,  quality  and  usage  metadata. 


Data  quality  includes  the  correctness,  timeliness,  accuracy,  completeness,  relevance,  and  accessibility  that  makes  data  appropriate  for 
use.  Quality  statements  are  required  for  source,  accuracy  (e.g.,  positional  and  attribute),  up-to-dateness  /  currency,  logical  consistency., 
completeness  (e.g.,  feature  and  attribute),  clipping  indicator,  security  classification,  and  releasibility  [DoD95]. 
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A  data  verification  and  validation  process  assesses  the  quality  of  the  data,  employing  data 
quality  measures,  and  establishes  the  basis  of  confidence  in  the  data  [Kil02], 

For  each  phase  or  aspect  of  the  simulation’s  design,  development  and  operation, 
possible  sources  of  performance  information  include  live  tests,  data  centers,  ground  tests  or 
analytical  M&S.  Theoretically,  the  data  producer  uses  metadata  to  describe  the  quality  of 
the  data  as  a  part  of  data  V&V  process  to  meet  the  specification  requirements  and  provides 
data  meeting  the  user’s  specification.  The  user  assesses  the  applicability  and  quality  of  the 
data  for  a  specific  application  based  on  the  producer’s  verified  metadata.  As  part  of  the 
users  V&V  process,  the  data  verification  process  checks  the  model  algorithms,  and  as  the 
last  step,  validates  the  data  and  algorithms  to  establish  overall  M&S  credibility.  The 
DMSO-sponsored  Authoritative  Data  Source  (ADS)  project  [VED98]  identified  seven 
hundred  and  fifty-six  authoritative  data  sources  that  met  ADS  data  procedures,  including 
metadata  definitions,  in  an  effort  to  decrease  the  redundancy  of  data  development,  and  im¬ 
prove  data  quality,  fidelity,  and  data  sharing. 

Environment  data  is  critical  to  the  M&S  community.  Sun  Tzu  realized  the  impor¬ 
tance  of  knowing  the  environment  when  he  wrote,  “Know  the  other,  know  yourself  and  the 
victory  will  not  be  at  risk;  know  the  ground,  know  the  natural  conditions,  and  victory  can 
be  total”  [Tzu71].  The  second  [DoD95]  objective  included  the  development  of  natural  en¬ 
vironment  representations  sub-objectives  for  terrain  domains  including:  (2.1),  ocean  (2.2), 
atmosphere  (2.3),  and  space  (2.4)  instantiated  by  the  Ocean,  Atmosphere  and  Space  Envi¬ 
ronmental  Services  (OASES)  System  [RIOOl], 

However,  quality  environmental  data  is  hard  to  acquire.  [BPT96]  studied  the  eight 
most-needed  types  of  environmental  data  identified  by  earlier  surveys  and  concluded  that 
the  capability  to  acquire  these  data,  especially  at  high  fidelity  were  insufficient  to  meet  re¬ 
quirements,  especially  in  the  models  of  fog,  aerosols,  humidity,  visibility,  and  databases. 
Responding  to  these  concerns,  the  Department  invested  heavily  in  environmental  data.  In 
an  early  DMSO  sponsored  initiative  to  improve  environmental  representations,  the  Naval 
Research  Laboratory  [NRL95]  developed  the  Environmental  Effects  for  Distributed  Inter- 
active  Simulation  (E  DIS)  program.  The  E  DIS  program  employed  an  object-oriented 


126 


methodology  described  by  [HJG94,  Hec94,  WH94,  AFW+94]  to  support  distributed  inter¬ 
active  M&S  with  environments  and  environmental  effects. 

The  Synthetic  Environment  Data  Representation  and  Interchange  Specification 
(SEDRIS)  program  is  a  major  Department  follow-on  initiative  to  the  E“DIS  program  with 
objectives  cited  by  [SED98a,  SED98b,  SED98c]  to  improve  environment  representations. 
The  SEDRIS  vision  [SED98a]  is  to  provide  a  common  synthetic  environment  to  reduce 
cost,  improve  reuse,  support  interoperability,  and  provide  effective  data  transfer  between 
heterogeneous  systems,  initially  in  the  training  functional  area.  SEDRIS  employed  a  meta¬ 
model  based  transmittal  methodology  with  a  standardized  data  model  and  supporting  appli¬ 
cation  program  interfaces.  The  SEDRIS  Data  Model,  based  on  an  Object  Model  Template 
(OMT),  with  minor  extensions,  employs  the  concept  of  a  class  of  data  [SED98a]  supporting 
a  common  environmental  architecture. 

SEDRIS’  major  function  is  to  serve  as  an  interchange  medium  supporting  diverse 
requirements  for  different  customer  domains  [Bir97].  SEDRIS  also  includes  a  Geospatial 
Reference  Model  (ISO  18024)  supporting  the  specification  of  coordinates,  datum,  projec¬ 
tions,  and  several  geo-  and  non-geo-referenced  spatial  reference  systems,  the  SEDRIS 
transmittal  fonnat  (ISO  1 8026)  and  the  Environmental  Data  Coding  Specification  (ISO 
180230)  [RMJ+02,  JMP02],  Complementing  these  initiatives  are  current  or  projected  ef¬ 
forts  to  acquire  science  data  from  other  the  private  sector,  international  scientific  programs, 
or  cooperative  public-private  partnerships. 

NASA  led  the  effort  to  collect  data  from  space-based  sensors,  and  continues  to  this 
day  developing  spacecraft  for  a  complete  Earth  Observing  System,  with  planned  experi¬ 
mental  Pathfinder  missions  under  the  auspices  of  the  Earth  Science  Enterprise  [PFL+00]. 
Another  emerging  international  initiative  collaborative  effort  promoting  efficient  geospatial 
data  development,  sharing  and  use  is  the  Global  Spatial  Data  Infrastructure  [LWK+02]. 

However,  remotely  sensed  data  from  space  [JBS+93,  NAB+92,  FG99]  may  be  dif¬ 
ficult  to  share  based  on  different  sensor  characteristics  (e.g.,  sensitivity,  spectral  coverage, 
calibration),  the  lack  of  standards,  proprietary  data  processing  systems  and  algorithms,  and 
subtle  variations  that  may  adversely  influence  data  consistency  [JSK02],  In  addition,  en¬ 
suring  commercial  data  sources  are  available  requires  an  understanding  of  negotiations  for 
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intellectual  property  [IEE99b,  ATL01]  and  understanding  of  how  information  technology 

1  T? 

standards  emerge,  survive  and  evolve  [LibOl], 


D.  SUMMARY 

Chapter  V  provided  the  Department’s  current  architectural  framework  for  resolving 
the  underlying  systemic  causes  generating  the  pre-conditions  for  heterogeneous  system  rep¬ 
resentation  anomalies,  especially  in  federation  interoperability.  The  Department’s  initia¬ 
tives  to  improve  the  credibility  of  large-scale,  legacy  Department  M&S  supporting  distrib¬ 
uted  interoperability  include  the  “as-is”  architecture  and  the  evolving  “to-be”  architecture: 
the  Joint  Technical  Architecture,  and  the  Common  Technical  Framework.  The  Common 
Technical  Framework  components  include  the  High-Level  Architecture,  conceptual  model 
requirements,  and  data  standards  supporting  credible  simulation  development. 

The  Department  established  tenns  of  reference  for  an  M&S  framework  to  improve 
the  communication  of  concepts,  interoperability,  and  development  of  authoritative  repre¬ 
sentations  including  architectural  implementations  (e.g.,  SIMNET,  ALSP,  DIS,  and  HLA). 
The  Federation  Development  and  Execution  Process  (FEDEP)  support  the  implementation 
of  the  HLA. 

The  Department’s  future  “to-be”  architecture  development  includes  the  evolving 
Joint  Technical  Architecture  (JTA),  common  Operating  Environment  (COE)  and  C4ISR 
Framework  supported  by  the  Technical  Reference  Model  (TRM)  with  a  major  focus  to  im¬ 
prove  interoperability  and  information  exchange.  The  COE,  JTA,  and  C4ISR  framework 
support  the  foundation  of  the  software-intensive  Department’s  To-Be  Architecture. 

The  Department’s  Modeling  and  Simulation  Domain  prescribed  by  [TRMOla]  in¬ 
cludes  the  Common  Technical  Framework  (CTF)  [DoD95,  GMS+96]  for  facilitating  M&S 
interoperability  and  reuse.  The  CTF  has  three  components:  the  High-Level  Architecture 
(HLA)  to  which  the  Department’s  which  M&S  must  conform;  Conceptual  Model  of  the 
Mission  Space  (CMMS)  to  provide  a  basis  for  the  development  of  consistent  and  authorita¬ 
tive  representations;  and  data  standards  to  provide  common  representations  of  data  across 
M&S  and  C4I  systems. 
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Information  technology  standards  are  a  means  by  which  two  or  more  products  (or  systems)  can  function  together.  [LibO  1  ] 
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A  conceptual  model  is  the  developer’s  method  of  translating  the  M&S  requirements 
into  a  detailed  design  framework  to  develop  the  software,  and  describes  the  conceptual 
model  components,  interactions,  and  the  M&S  concept  of  operations.  [SPC+98,  PacOOa, 
SHOO,  SheOOb,  DDH01]  validated  the  basic  need  for  conceptual  models,  and  developed 
supporting  perspectives  of  conceptual  modeling  theory,  identified  systemic  issues,  and  pro¬ 
posed  solutions.  However,  the  Department’s  M&S  community  currently  lacks  consensus 
and  standards  for  conceptual  models,  which  are  critical  to  the  key  Department’s  M&S  is¬ 
sues  of  authoritative  representations,  improved  simulation  credibility,  enhanced  confidence 
in  simulation  results,  integrated  simulation  security,  and  improved  simulation  interoperabil¬ 
ity. 

Surveying  the  current  M&S  situation,  [LRH+01]  consider  the  term  “conceptual 
model”  extremely  overloaded  and  overused  within  the  M&S  community.  For  example,  the 
term  “conceptual  model”  describes  the  first  abstraction  of  representations  in  an  M&S  and 
describes  a  high-level  design  of  how  all  the  components  in  an  M&S  relate  to  another.  The 
wide  and  varied  body  of  literature  on  the  subject  of  conceptual  models  is  an  indication  of 
their  importance,  and  also  reveals  a  lack  of  consensus  on  conceptual  model  methods,  veri¬ 
fication  and  validation,  fonnat  and  development  methodology. 

The  term  Functional  Description  of  the  Mission  Space  (FDMS)  replaced  the  previ¬ 
ous  Conceptual  Model  of  the  Mission  Space  (CMMS)  term,  in  the  Department’s  M&S 
lexicon,  as  a  required  component  of  the  M&S  development  and  V&V  processes.  The 
FDMS  provides  the  real-world  description  of  entities,  processes,  and  activities  for  the  de¬ 
sign  phase,  and  detailed  representations  of  the  problem  domain  in  the  requirements  devel¬ 
opment  phase  of  the  simulation  life  cycle. 

Data  standards  are  the  third  component  of  the  common  technical  framework  for 
M&S.  In  any  M&S  application,  the  associated  data  quality  should  be  as  creditable  as  the 
M&S  itself  if  the  user  or  sponsor  is  to  attach  confidence  to  the  results.  [DoD95]  defines 
data  quality  for  the  Department’s  M&S  programs.  [BCE+99]  also  suggests  data  quality 
attributes  including  source  accuracy,  fidelity,  suitability,  credibility,  maturity,  cost,  avail¬ 
ability,  and  similar  characteristics.  [Kil02]  recommends  data  accuracy  and  quality  informa- 
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tion  supported  by  a  review  at  three  levels:  database,  data  element,  and  data  value;  employ¬ 
ing  descriptive,  quality  and  usage  metadata. 

Theoretically,  the  data  producer  uses  metadata  to  describe  the  quality  of  the  data  as 
a  part  of  data  V&V  process  to  meet  the  specification  requirements  and  provides  data  meet¬ 
ing  the  user’s  specification.  The  user  assesses  the  applicability  and  quality  of  the  data  for  a 
specific  application  based  on  the  producer’s  verified  metadata.  As  part  of  the  users  V&V 
process,  the  data  verification  process  checks  the  model  algorithms,  and  as  the  last  step, 
validates  the  data  and  algorithms  to  establish  overall  M&S  credibility. 

Environment  data  is  critical  to  the  M&S  community.  However,  quality  environ¬ 
mental  data  is  hard  to  acquire.  In  an  early  DMSO  sponsored  initiative  to  improve  envi¬ 
ronmental  representations,  the  Naval  Research  Laboratory  developed  the  Environmental 
Effects  for  Distributed  Interactive  Simulation  (E"DIS)  program.  The  Synthetic  Environ¬ 
ment  Data  Representation  and  Interchange  Specification  (SEDRIS)  program  is  a  major  De¬ 
partment  to  improve  environment  representations.  The  SEDRIS  vision  is  to  provide  a 
common  synthetic  environment  to  reduce  cost,  improve  reuse,  support  interoperability,  and 
provide  effective  data  transfer  between  heterogeneous  systems,  initially  in  the  training 
functional  area.  The  SEDRIS  program  employs  a  meta-model  based  transmittal  methodol¬ 
ogy  with  a  standardized  data  model  and  supporting  application  program  interfaces. 

Complementing  these  initiatives  are  current  or  projected  efforts  to  acquire  science 
data  from  other  the  private  sector,  international  scientific  programs,  or  cooperative  public- 
private  partnerships.  However,  remotely  sensed  data  from  space  may  be  difficult  to  share 
based  on  different  sensor  characteristics  (e.g.,  sensitivity,  spectral  coverage,  calibration), 
the  lack  of  standards,  proprietary  data  processing  systems  and  algorithms,  and  subtle  varia¬ 
tions  that  may  adversely  influence  data  consistency.  In  addition,  ensuring  commercial  data 
sources  are  available  requires  an  understanding  of  negotiations  for  intellectual  property. 
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VI.  MODEL  QUALITY  ATTRIBUTES  LOR  IMPROVED  CREDIBILITY 

A.  INTRODUCTION 

Chapter  VI  addresses  software  and  simulation  model  quality  attributes  affecting 
heterogeneous  system  representation  anomalies  and  credibility  in  Department  M&S,  espe¬ 
cially  in  federation  interoperability.  The  chapter  reviews  the  internal  and  external  attributes 
of  the  ISO  9126-1  Quality  model  [ISOOl],  testing  for  quality  attributes,  and  M&S  quality 
attributes  addressed  in  much  of  the  current  literature,  including  an  overview  of  aggregation 
and  disaggregation.  The  chapter  concludes  with  a  discussion  on  M&S  quality  approaches. 

B.  SYSTEMIC  SIMULATION  SOFTWARE  QUALITY  ISSUES 

In  the  1970s,  the  cost  of  software  maintenance  for  the  first  time  exceeded  the  cost 
of  software  development,  creating  a  “software  crisis”  and  identifying  for  the  first  time  that 
the  technology  at  the  time  was  inadequate  to  develop  large,  complex  software-based  sys¬ 
tems.  By  the  mid-1980s  “silver  bullet”  solutions  such  as  computer-aided  software  engi¬ 
neering  (CASE)  tools  were  tried  and  found  wanting  to  solve  the  “software  crisis”  as  soft¬ 
ware  evolved  into  a  $300  billion  dollar  a  year  industry  [RakOl]. 

In  1988,  the  Department’s  Director  of  Operational  Test  and  Evaluation  (OT&E) 
[Kri88]  acknowledged  that  the  evolving  art  of  M&S  needed  guidance  to  improve  quality 
and  establish  credibility  of  simulation  results  through  verification,  validation  and  credibility 
assessments.  [Kri89]  implemented  a  policy  on  M&S  support  of  OT&E  specified  by 
[DOT89]  that  identified  the  following  high-level  objectives  in  the  model  development 
process  supporting  credibility,  improved  quality  and  confidence  in  M&S  results: 

•  Acceptability  of  the  M&S  approach, 

•  Confidence  in  the  model, 

•  Confidence  in  the  M&S  team, 

•  Confidence  in  methodology  and  use, 

•  M&S  verification,  validation  and  accreditation  [DOT89]. 

Society’s  growing  dependency  on  large,  complex  software-intensive  systems  com¬ 
plicates  the  development  of  quality  software.  According  to  [BooOl],  as  “software  contin¬ 
ues  to  weave  itself  deeply  into  the  fabric  of  society,  the  stakes  have  gotten  higher”  [BooOl], 


131 


This  challenge  has  driven  the  development  of  many  different  methods,  techniques  and  ap¬ 
proaches  to  improve  software  quality  supporting  the  Department’s  large-scale,  software¬ 
intensive,  simulation  systems. 

C.  SOFTWARE  QUALITY  INITIATIVES 

Today,  Department  testers  still  require  sufficient  confidence  in  the  quality  of  M&S 
results  [DOT99,  COHOO],  to  support  a  foundation  of  trust  for  using  M&S  results  as  the  ba¬ 
sis  of  management  decisions  or  part  of  the  fonnal  test  and  evaluation  process.  The  De¬ 
partment  initiated  several  programs  to  improve  the  relationship  between  M&S  and  live  test¬ 
ing,  reduce  costs  and  improve  the  overall  testing  process.  In  1995,  the  Secretary  of  De¬ 
fense  approved  live  initiatives,  including  the  more  effective  use  of  M&S,  to  mature  the 
OT&E  community  culture  from  a  pass/fail,  event-driven  paradigm  to  the  view  of  testing  as 
a  learning  process. 

[Wai02b]  discusses  several  of  these  initiatives,  including  the  role  of  hardware-in- 
the-loop  M&S  in  missile  defense  systems  testing  and  the  emerging  collaborative  testing 
environments  developing  in  technical  testing  architectures  such  as  the  Test  and  Training 
Enabling  Architecture  (TENA)  [PLL+02]  and  the  Virtual  Proving  Ground  (VPG).  In  an 
effort  to  improve  M&S  management  in  the  Department’s  test  and  evaluation  process 
[Obr99]  proposed  the  Modeling  and  Simulation  Test  and  Evaluation  Reform  (MASTER) 
initiative,  based  on  specific  categories  of  expertise  (e.g.,  terrain,  weather),  etc  called  M&S 
vectors.  The  MASTER  concept  [Obr99]  also  supported  verification  and  validation  of  as¬ 
signed  models  performed  by  government  R&D  centers,  and  a  consortium  organizational 
structure  to  implement  policy  and  maintain  a  repository  of  codes. 

Quality  is  an  essential  attribute  supporting  the  safe  operation  of  M&S  software  sys¬ 
tems,  especially  M&S  involving  possible  life  or  death  situations  [Bow02].  [LevOO]  de¬ 
scribed  the  root  causes  found  in  the  software  industry’s  flawed  quality  processes,  which 
created  the  conditions  for  the  Ariane  5  failure  in  1 996  resulting  in  an  uninsured  five  billion 
dollar  loss.  Several  major  spacecraft  losses  including  NASA’s  Mars  Polar  Lander,  Mars 
Global  Surveyor,  and  the  Titan  IV  /  MILSTAR  spacecraft  [LevOla],  revealed  similar  soft¬ 
ware  engineering  quality  shortfalls.  It  is  probable  that  similar  software  engineering  quality 
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and  safety  issues  exist  within  the  Department’s  software  engineering  programs  today  ac¬ 
cording  to  [HNC+00].  Furthermore,  [HNC+00]  found  the  current  level  of  the  Depart¬ 
ment’s  software  engineering  practice  lacks  discipline  and  consistent  enforcement  mecha¬ 
nisms  necessary  to  improve  these  conditions. 

There  have  been  several  uses  and  revisions  of  the  tenn  software  quality133  including 
[MRW77,  IEE89,  IEE92,  IEE98d]  over  the  past  twenty-five  years.  Today,  software  quality 
remains  open  to  subjectivity,  different  views,  and  various  interpretations  of  definitions. 
[MRW77]  addressed  software  quality  attributes  in  an  early  effort  and  proposed  a  software 
quality  model  that  identified  three  categories  of  software  quality  attributes.  The  study  of 
software  quality  and  software  quality  metrics  [IEE89,  IEE98d]  evolved  continuously  with 
the  maturation  of  software  engineering  practices  and  the  growth  of  the  software  industry. 

The  ISO  developed  a  two-part  software  quality  specification  model  [ISOOl],  pro¬ 
vided  at  Figure  6-1,  applicable  to  every  type  of  software  product.  The  current  [ISOOl] 
definition  of  quality,  similar  to  the  IEEE  definition  cites  “the  totality  of  characteristics  of  an 
entity  that  bare  on  its  ability  to  satisfy  stated  and  implied  needs”  [ISOOl].  The  ISO  Quality 
Model  identifies  six  characteristics  for  internal  and  external  product  quality  and  their  asso¬ 
ciated  sub  characteristics.  An  additional  four  quality- in-use  characteristics  prescribed  by 
[ISOOl]  describes  the  combined  effects  of  internal  and  external  product  quality. 

The  study  of  software  quality  by  [Dro95,  Dro96]  suggests  that  the  emphasis  on 
quality  software  development  has  been  heavily  process-oriented  and  proposes  a  product- 
based  model  composed  of  components  and  modules  possessing  three  key  quality  proper¬ 
ties:  cohesion,  coupling,  and  layering134,  linked  to  quality  attributes.  As  object-oriented 
(00)  languages  continue  to  evolve,  [AleOl]  also  addressed  the  additional  complexity  of 
00  languages  over  previous  procedural  programming  testing  techniques  in  00  testing,  in¬ 
cluding  inheritance,  polymorphism  and  complex  data  requiring  adequate  testing  of  all  rela¬ 
tionships.  [FowOl]  suggests  the  earliest  indication  of  design  quality  is  the  level  of  cou¬ 
pling,  high-level  dependencies  and  cohesion. 


Software  quality  is  the  degree  to  which  software  possess  a  desired  combination  of  quality  attributes  [IEE98d]. 

134  ... 

Layering  for  objects,  programs,  processes  and  systems  are  governed  by  the  constructive  principle  that  components  constructed  at  one 
level  may  only  be  used  at  the  next  higher  level.  [Dro96] 
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Quality  Model 
for  Internal  and 
External  Quality 

QUALITY  CHARACTERISTICS  (6) 


Functionality 

Reliability 

Usability 

Efficiency 

M  aintainability 

Portability 

•Suitability 

•M  aturity 

•Understand  ability 

•Tim  e 

•Analyzability 

•Adaptability 

•Accuracy 

•Fault 

•Learnability 
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•Changeability 

•Installability 

•Interoperability 
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•Operability 

•Resource 

•Stability 
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•Security 

•Recoverability 

•Attractiveness 
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•T  estability 

•Replaceability 
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com  pliance 

com  pliance 

com  pliance 

Note  1:  There  are  Compliance  SuhCharacteristics  for  all  Characteristics,  with  generally  applicable  principles 


Quality 

In  U  se 


QUALITY  IN  USE  CHARACTERISTICS  (4) 


Effectiveness  Productivity  Safety  Satisfaction 


Figure  6-1.  The  ISO  9126-1  Quality  Model  (From  [ISOO 1  ]) 


[IEE98d]  provides  a  standard  for  software  quality  metrics  supporting  assessments 
for  both  the  process  and  the  product  on  the  status  of  meeting  quality  requirements  through¬ 
out  the  software  life  cycle.  [Sch02a]  incorporates  quality  metrics  into  software  engineering 
practices,  citing  a  growing  quality  measurement  body  of  knowledge  [IEEOOa]  augmenting 
the  IEEE  software  quality  standard.  [BD02]  added  an  extension  to  the  [Dro96]  hierarchical 
methodology  quality  framework  work  of  developing  the  Quality  Model  for  Object-oriented 
Design  (QMOOD)  to  improve  object-oriented  design  quality.  [BD02]  based  the  QMOOD 
model  on  the  following  broad  quality  attributes:  functionality,  effectiveness,  understand- 
ability,  extendibility,  reusability,  and  flexibility,  in  lieu  of  the  ISO  9126  quality  attributes 
[ISOOl]  listed  in  Figure  6-1. 

Methodologies  for  improved  software  quality  continue  to  appear.  [BHK01]  sug¬ 
gests  software  documentation  inspections  in  the  early  stages  of  a  software  project  may  al¬ 
low  for  the  early  identification  of  defects,  and  experiments  conducted  by  [BHK01]  support 
the  employment  of  a  second  inspection  cycle  to  improve  overall  quality.  [HJB+98]  took  a 
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different  approach  and  developed  a  method  for  developing  product/process  dependency 
models  (PPDMs),  empirical  software  artifacts,  reflecting  past  experiences  to  improve  the 
planning,  techniques,  tools  or  methods  for  future  projects.  In  addition,  [FPC97,  KisOl] 
identified  measurement  models  and  frameworks  by  supporting  the  software  engineer’s  abil¬ 
ity  to  establish  product  measures  supporting  improved  overall  software  quality  and  reliabil¬ 
ity.  [BooOl]  suggests  properly  applied  measures  provide  management  with  important  in¬ 
sights  into  the  health  and  welfare  of  a  software  project. 

1.  Internal  /  External  Quality 

[ISOOl]  divides  software  quality  into  six  major  characteristics:  functionality,  reli¬ 
ability,  usability,  efficiency,  maintainability,  and  portability,  with  each  characteristic  com- 

1  o  c 

posed  of  sub-characteristics  .  Functionality  is  the  capability  of  the  software  product 
functions  to  meet  stated  and  implied  needs  under  certain  specified  conditions,  and  includes 
the  sub-characteristics  of  suitability  ,  accuracy,  interoperability,  and  security. 

The  [DoD98]  glossary  defines  accuracy  as  “the  degree  of  exactness  of  a  model  or 
simulation,  high  accuracy  implying  low  error”  .  In  addition,  “accuracy  equates  to  the 
quality  of  the  result,  and  is  distinguished  from  precision”  [DoD98].  A  significantly  differ¬ 
ent  definition  of  accuracy  by  [RGHOO]  cites  accuracy  as,  “the  agreement  between  the  per¬ 
formance  of  these  models  of  each  aspect  and  the  real  world  performance”  [RGHOO].  An 
additional  definition  from  the  [RPGOO]  guidelines  defines  accuracy  as  the  “degree  parame¬ 
ters,  parameter  sets,  or  variables  correspond  exactly  to  the  simulation  reality,  referent  or 
some  chosen  standard”  [RPGOO] . 

Accuracy  is  also  one  of  several  model  attributes  and  dimensions  related  to  model 
real-world  fidelity,  and  a  quality  sub-characteristic  of  software  functionality  identified  by 
[ISOOl]  as  the  capability  to  achieve  the  agreed  upon  or  correct  solutions  with  the  required 


135  There  are  compliance  sub-characteristics  for  all  six  characteristics  with  similar  standards  and  protocols  [ISOOl]. 

136  Suitability  is  the  software’s  fitness  of  purpose  to  provide  an  appropriate  set  of  functions  to  complete  certain  tasks,  meet  user  objec¬ 
tives,  and  affects  operability  [ISOOl]. 
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Error:  The  difference  between  an  observed,  measured,  or  calculated  value  and  a  correct  value  [RPGOO]. 
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degree  of  precision1 18.  According  to  [Kil02]  confidence  in  software  quality  and  accuracy 
depends  on  the  development  environment,  verification  evidence,  and  software  quality  as¬ 
surance  [IEE89]  and  assessments,  which  includes  accepted  software  development  practices, 
personnel  skill  and  experience  attributes,  sufficient  resources  and  the  existence  of  key  life 
cycle  development  artifacts  such  as  configuration  management  histories  and  logs.  [Kil02] 
also  notes,  “dedicated  software  engineering  experience  is  essential  to  most  large-scale 
software  development  efforts”  [Kil02] . 

1  '7.Q 

M&S  interoperability  is  also  an  extremely  complex  systemic  issue,  and  one  of  the 
central  aspects  of  the  Department’s  current  M&S  efforts.  Major  M&S  interoperability  is¬ 
sues  confronting  joint  model  development  and  federations  include  data,  algorithms 
[CLR97],  representations,  and  joint  C3I  [You97,  DPB+01].  [ISOOl]  defines  interoperabil¬ 
ity  as  the  ability  of  the  software  to  interact  with  specified  systems.  However,  interoperabil¬ 
ity  is  not  just  an  M&S  specific  challenge,  it  is  a  universal  Department  issue,  which  impacts 
many  sectors  of  the  Department  information  domain  [Ngu95,  Wof95,  HKL96,  AH97, 
HMD99,  Sut99,  DSB+99,  HamOOb,  HDGOO,  YBG+01,  HMOO,  GBL01,  LBG+01, 
WHL+01,  BKW+02,  CW02].  Software  developers  and  maintainers  may  introduce  inter¬ 
operability  anomalies  in  autonomously  developed  software-intensive  systems  in  all  phases 
of  a  systems  life  cycle. 

Interoperability  is  also  a  multi-dimensioned  challenge.  The  interoperability  re¬ 
quirements  listed  by  [DoD02d]  are  similar  to  the  architecture  concept  for  seamless  M&S 
with  a  single  integrated  environment  proposed  by  [Dow92],  The  synthetic  environment 
ideas  advanced  by  [Dow92]  identifies  three  classes  of  requirements:  1)  simulation  truth,  2) 
conceptual  consistency,  and  3)  temporal  consistency,  which  generate  the  supporting  techni¬ 
cal  dimensions  including  the  a)  system  architecture,  b)  data  management,  c)  human  ma¬ 
chine  interaction,  and  d)  time  management.  [Ham96,  HNP97]  and  [HM02]  address  the 
management  of  time,  clocks,  and  synchronization  in  distributed  simulation,  the  key  role  of 


138  Precision  relates  to  the  quality  of  the  operation  by  which  the  result  is  obtained  and  can  be  repeated,  and  is  distinguished  from  the  term 

accuracy  that  is  the  quality  of  the  result.  [DoD98] 

139 

M&S  interoperability  is  the  ability  of  a  model  or  M&S  to  provide  services  to  and  accept  services  from  other  M&S,  and  to  use  the 
services  so  exchanged  to  operate  effectively  together  [DoD98] 
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partitioning,  and  identifies  five  methods  for  partitioning  a  simulation  for  distributed  execu¬ 
tion,  with  synchronization  noted  as  the  most  important  implementation  consideration. 

The  Internet  is  a  major  new  interoperability  dimension.  With  the  explosive  growth 
in  the  Internet  in  the  1990’s,  the  Clinton  Administration  conceived  a  National  Information 
Infrastructure  (Nil)  program  to  deliver  to  all  Americans  the  information  they  needed  at  an 
affordable  price.  A  complementary  Department  initiative  effort  to  the  Nil,  the  Defense  In¬ 
formation  Infrastructure  (DII)  [TM97],  provides  support  for  “dual-use”  technology  and 
added  privacy,  reliability,  and  security  quality  attributes  to  Nil  initiatives.  The  DII  founda¬ 
tion  elements140  [TM97]  included  modeling  and  simulation,  standards,  architectures,  soft¬ 
ware  engineering,  and  test  and  evaluation. 

However,  [SFJ+98]  cited  continuing  interoperability  issues  at  almost  every  level. 
The  Department  detennined  a  need  to  develop  better  policies  and  procedures  to  align  and 
resource  the  technology  base  to  support  infonnation  assurance  [DoD02e],  information  dis¬ 
semination  management,  interoperability,  network  management,  network  operations,  and 
enterprise  computing  and  implemented  the  Global  Information  Grid141  (GIG)  and  the  GIG 
Architecture142  (GIGA)  [GIGOO].  [HamOOb]  provided  Departmental  GIG  policy  and  guid¬ 
ance.  [DPB+01,  ChaOl,  WolOl,  BBN+01]  also  identified  the  need  for  new  analytical  tools 
to  support  improved  interoperability  and  facilitate  the  industrial  age  to  information  age 
transfonnation  process. 

The  security  quality  attribute143  addresses  the  ability  of  the  software  to  prevent  un¬ 
authorized  access  to  programs  or  data,  whether  accidental  or  deliberate  [ISOO 1  ] .  The  Na¬ 
tional  and  the  Departmental  strategies  to  counter  threats  and  security  challenges 
[WHM+79,  LX99,  KH02,  SB02a]  include  natural  environment,  man-made  physical  haz¬ 
ards,  and  human  actors,  which  are  now  considered  more  complex  and  more  difficult  than 
earlier  threats  [WHM+79].  The  complex  global  interdependent  infonnation  environment 
identified  by  the  worldwide  Year  2000  remediation  effort  [Gre97,  WG99,  Mus02]  includes 

140  DII  foundation  elements  included  policy,  requirements,  modeling  and  M&S,  standards,  architectures,  technology  base,  software  engi¬ 
neering,  test  and  evaluation,  and  joint  spectrum  management  [TM97]. 

141 

The  GIG  is  “a  globally  interconnected  end-to-end  set  of  information  capabilities,  associated  processes  and  personnel  for  collecting, 

storing,  disseminating,  and  managing  information  on  demand  to  warfighters,  policy  makers,  and  support  personnel”  [HamOO,  GIGOO]. 

142 

The  GIGA  is  composed  of  interrelated  operational,  systems,  and  technical  views,  which  defines  the  characteristics  of  and  relation¬ 
ships  among  current  and  planned  Global  Information  Grid  assets  in  support  of  National  Security  missions  [GIGOO]. 

143 

Security  quality  attributes  are:  availability,  identification,  authentication,  confidentiality,  integrity,  and  non-repudiation  [WooOOb]. 
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critical  infrastructure144  protection  [EFL+97,  ABBOO,  BusOl,  GPH+01],  information  opera¬ 
tions  (10) 145  doctrine  [JP98],  10  opportunities  and  vulnerabilities  addressed  by  [SBJ+02], 
and  information  superiority  (IS)146  issues  [JP98]. 

The  Department  is  just  beginning  to  understand  the  risks  to  an  information  superior¬ 
ity  strategy  and  infonnation  operations  (10)  as  a  basis  for  safeguarding  the  Nation’s  critical 
interests,  and  responding  with  information  assurance  (IA)  measures.  Information  assur¬ 
ance147  includes  risks  and  protective  measures  [HNP97,  HamOOb,  ABBOO,  WooOO, 
MonOOb,  WFP+01,  DoD02d,  DoD02e].  The  Department  selected  Millennium  Challenge 
02  (MC02),  a  large-scale  experiment  to  define  the  impact  of  10  measures  employing  the 
Joint  Staff  Analysis  Model,  as  part  of  a  limited-objective  IO/IA  experiment  [A1102] .  Initial 
MC02  results  suggest  that  modeling  advanced  10,  IS,  and  IA  capabilities  present  signifi¬ 
cant  challenges  to  the  Departmental  M&S  community. 

Reliability,  the  second  of  six  major  [ISOOl]  software  quality  characteristics,  in¬ 
volves  the  capability  of  software  used  under  certain  conditions  to  meet  a  specified  level  of 
performance.  Reliability  is  comprised  of  the  following  sub-characteristics:  maturity,  fault 
tolerance,  and  recoverability  [ISOOl],  Since  software  does  not  wear  out  or  age,  limitations 
in  reliability  are  rooted  in  the  requirements,  design,  implementation,  and  maintenance 
processes.  One  hundred  percent  software  reliability  may  be  an  impossible  goal  to  attain  in 

148 

the  current  software  development  environment  when  one  considers  that  even  ‘six-sigma’ 
strategies  seek  only  to  reduce  the  number  of  errors  in  software  systems. 

Software  reliability  engineering  (SRE)149  practices  are  critical  in  high-risk,  safety- 
critical  areas,  which  affect  national  security  or  risk  human  life.  NASA  Shuttle  mission  en¬ 
gineers  successfully  implemented  the  Space  Shuttle  Flight  Software  Application  methodol- 


144 

Critical  infrastructure  was  defined  in  the  1998  Presidential  Decision  Directive  63  as  “those  physical  and  cyber-based  information 

systems  essential  to  the  minimum  operations  of  the  economy  and  the  government”  [SBJ+02]. 

145 

Information  operations  are  operations  conducted  to  defend  our  own  information  and  information  systems  and  affect  adversary  infor¬ 
mation  and  information  systems  [WooOOb]. 

146 

Information  superiority  is  the  capability  to  collect,  process  and  disseminate  an  uninterrupted  flow  of  information,  while  exploiting  or 

denying  an  adversary’s  capability  to  do  the  same  [WooOOb]. 

147 

Information  assurance,  a  subset  of  information  operations,  is  actions  that  protect  and  defend  information  and  information  systems  by 
ensuring  availability,  integrity,  authentication,  confidentiality,  and  non-repudiation,  and  includes  restoration  of  information  systems  by 
incorporating  protection,  detection,  and  reaction  capabilities  [WooOOb]. 

148 

Six  Sigma  objectives  are  to  reduce  the  number  of  defects  to  3.4  defects  per  million  lines  of  code  [STSOO]. 

149 

SRE  is  the  application  of  statistical  techniques  to  data  collected  during  system  development  and  operation  to  specify,  predict,  esti¬ 
mate,  and  assess  the  reliability  of  software-based  systems.  [Sch99a] 
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ogy  [KS97,  Sch97a,  Sch97b,  Sch99a]  to  improve  overall  Shuttle  software  reliability. 
Software  reliability  affects  all  aspects  of  the  software  life  cycle  to  include  architecture 
[Lew02a],  requirements  [SchOla,  SchOlb],  risk  [SchOld],  software  maintenance  process 
[Sch99b],  commercial-off-the-shelf  software  use  [SchOOc],  client-server  systems  [Sch96], 
testing  [JDL01],  and  software  quality  control  and  prediction  [SchOOc],  [Rus98,  IC99, 
RC99,  IC01,  RC01]  describe  a  decision  support  tool  for  selecting  a  reliability  strategy 
based  on  project,  product  and  resource  decision  factors.  Other  software  reliability  sub¬ 
characteristics  include  maturity,  fault  tolerance,  and  recoverability  where: 

•  Maturity  is  the  ability  of  the  software  to  avoid  failure  stemming  from  faults  in  the 
software  product,  or  the  frequency  of  failures, 

•  Fault  tolerance  is  the  robustness  of  the  system  and  its  ability  to  maintain  a  specific 
level  of  perfonnance  in  the  event  of  faults  or  specified  interface  problems,  and  may 
include  a  fail-safe  capability, 

•  Recoverability  is  the  software’s  ability  to  regain  a  specified  level  of  performance 
and  recover  affected  data  in  the  event  of  a  failure  [ISOOl]. 

Usability  is  the  third  of  six  major  [ISOOl]  software  quality  characteristics.  Usabil¬ 
ity  includes  the  ability  of  a  user  to  understand,  leam,  and  employ  the  software  produce  un¬ 
der  specified  conditions  and  is  comprised  of  the  following  sub-characteristics:  understand¬ 
ing,  leamability,  operability,  and  attractiveness  [ISOOl]  where: 

•  Understandability  is  dependent  on  the  documentation  and  the  initial  impression  of 
the  software,  and  allows  the  user  to  detennine  if  the  software  is  suitable,  and  how  it 
may  be  used  for  specific  tasks  and  under  what  conditions, 

•  Leamability  is  the  ability  of  the  user  to  leam  to  how  to  use  the  application, 

•  Operability  is  the  ability  of  the  user  to  operate  and  control  the  software  product,  and 
is  affected  by  the  attributes  of  suitability,  changeability,  adaptability,  and  installabil- 

ity, 

•  Attractiveness  is  normally  associated  with  the  graphical  design,  uses  of  color,  and 
the  ability  of  the  software  to  be  attractive  to  the  user  [ISOOl]. 

The  usability  sub-characteristics  are  subjective,  dependent,  most  difficult  to  meas¬ 
ure,  and  are  the  least  objective  quality  factors.  [JWC01]  acknowledge  that  usability  is  a 
difficult  characteristic  to  integrate  into  any  system,  and  requires  specific  knowledge  of  user 
requirements,  preferences,  and  limitations.  Research  by  [GRS02]  borrowed  from  the  psy- 
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chology  field  in  studying  the  relationship  between  pain  research  and  the  implications  for 
usability  in  software  engineering,  specifically  the  peak-end  effect.150 

Research  of  simulation  software  usability  by  [Kil02]  suggests  that  simulation  soft¬ 
ware  is  credible  only  when  employed  within  a  well-defined  context  of  use;  that  usability  is 
more  a  case  of  reducing  the  probability  of  simulation  misuse  rather  than  the  ease  of  use; 
and  includes  the  M&S  user  support  activities,  which  facilitate  the  credible  use  of  the  M&S. 
[DoDOla]  found  most  Department  simulations  were  too  difficult  to  use  and  too  resource 
intensive  to  employ  as  widely  as  needed  to  achieve  the  complete  range  and  scope  of  re¬ 
quired  objectives.  [Kil02]  also  identified  configuration  management  as  an  indicator  of 
M&S  usability  and  the  “glue  that  ties  the  version  of  the  simulation  to  all  the  V&V  results 
and  M&S  documentation”  [Kil02] . 

Efficiency,  the  fourth  of  the  six  major  [ISOOl]  software  quality  characteristics,  in¬ 
volves  the  software’s  resource  dependent  ability  to  provide  appropriate  performance,  under 
stated  conditions.  Time-behavior  and  resource  utilization  comprise  the  [ISOOl]  sub¬ 
characteristics  of  efficiency  where: 

•  Time  behavior  allows  the  software  under  stated  conditions  to  provide  appropriate 
responses,  throughput  rates,  and  processing  times, 

•  Resource  utilization  is  the  ability  of  the  software,  under  prescribed  conditions  to 
perform  its  function  using  appropriate  amounts  and  types  of  software  [ISOOl]. 

The  fifth  of  six  major  [ISOOl]  software  quality  characteristics  is  maintainability. 
Maintainability  addresses  the  ability  to  modify  the  software  product,  including  corrections, 
improvements,  or  adaptation  of  the  software  to  changes  in  the  requirements,  environment 
or  functional  specifications,  and  includes  the  following  [ISOOl]  sub-characteristics  where: 

•  Analyzability  is  the  ability  to  diagnose  the  software  for  deficiencies,  failures,  or  for 
ease  of  maintenance  and  modification,. 

•  Changeability  incorporates  design,  coding,  and  documentation  and  the  ability  to 
implement  a  specific  modification, 

•  Stability  allows  the  software  to  overcome  unintended  effects  from  software  modifi¬ 
cations, 

•  Testability  provides  the  capability  to  validate  the  modified  software  [ISOOl]. 


150  Redelmeier  and  Kahneman  discovered  the  peak-and-end  effect  in  patient  study  research  when  patients  were  asked  to  rate  their  pain  at 
regular  intervals  during  the  procedure  [GRS02] . 
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Portability,  the  sixth  and  last  of  the  six  major  [ISOOl]  software  quality  characteris¬ 
tics  deals  with  the  sub-characteristics  of  adaptability,  installability,  co-existence,  and  re- 
placeability  to  support  standards  and  conventions  of  software  portability  from  one  envi¬ 
ronment  to  another. 

2.  Quality  in  Use 

The  [ISOOl]  standard  specifies  quality-in-use  characteristics  as  enablers  allowing 
users  to  satisfactorily,  effectively,  productively,  and  safely,  achieve  specified  objectives. 
The  [ISOOl]  quality-in-use  characteristics  include  effectiveness  and  productivity  where: 

•  Effectiveness  is  the  software’s  ability  to  accurately  and  completely  achieve  speci¬ 
fied  goals, 

•  Productivity  is  the  software’s  ability  to  achieve  results  effectively  when  compared 
to  the  expended  resources  [ISOOl], 

Technical  constraints  include  the  lack  of  tools,  data  security,  data  descriptions,  variable 
resolutions  [HOB95],  and  hardware/software  limitations.  Cultural  challenges  include  the 
lack  of  trained  personnel,  immature  processes  and  the  lack  of  acceptance  of  M&S  tools, 

Safety  is  the  software’s  ability  to  maintain  acceptable  levels  of  risk  to  people,  envi¬ 
ronment,  property,  business,  or  software  [ISOOl].  [LevOO,  LevOla,  LevOlb,  LevOlc]  iden¬ 
tified  the  causal  factors  related  to  software  safety  issues  in  recent  air  and  space  accidents 
including  the  1996  explosion  of  the  Ariane  5  launcher,  the  1999  loss  of  the  Mars  Climate 
Orbiter,  the  placement  of  a  MILSTAR  satellite  in  an  incorrect  and  unusable  orbit  in  1999, 
and  the  destruction  of  the  Mars  Polar  Lander  in  2000.  Several  additional  space  accident 
reports  [LLF+96,  SMB+99,  YAB+00]  and  aircraft  accident  reports  [Lad93,  SL94,  Lad95] 
revealed  very  similar  systemic  safety  issue  factors  including,  but  not  limited  to: 

•  Overconfidence  and  over-reliance  on  digital  automation,  where  software  complex¬ 
ity  is  underestimated  and  the  effectiveness  of  testing  is  overestimated, 

•  Not  understanding  the  risks  associated  with  software, 

•  Confusing  reliability  with  safety, 

•  Over-reliance  on  redundancy, 

•  Assuming  risks  decrease  over  time, 

•  Ignoring  warning  signals, 

•  Inadequate  cognitive  engineering, 

•  Inadequate  specifications, 
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•  Flawed  review  processes, 

•  Inadequate  system  safety  engineering, 

•  Violation  of  basic  safety  engineering  practices  in  the  digital  parts  of  the  system, 

•  Software  reuse  without  appropriate  safety  analysis, 

•  Unnecessary  complexity  and  software  functions, 

•  Operational  personnel  not  understanding  automation, 

•  Test  and  simulation  environments  that  do  not  match  the  operational  environment, 

•  Deficiencies  in  safety-related  infonnation  collection  and  uses  [LevOlb]. 

[LevOla]  expanded  upon  the  cited  accident-causal  factors  and  noted  their  existence 
in  three  systemic  organizational  categories  including  flaws  in  the  safety  culture,  ineffective 
organizational  structure  and  communication,  and  inadequate  or  ineffective  technical  activi¬ 
ties.  In  aerospace  systems,  a  general  testing  heuristic  is  to  fly  what  you  test  and  test  what 
you  fly.  However,  [LevOlb]  noted  in  the  Ariane  5  case  that  developers  experienced  all 
three  accident-causal  factors  and  violated  the  general  testing  heuristic  when  they  reused  the 
trajectory  data  of  the  Ariane  4  in  the  simulation  and  specification  of  the  Ariane  5  software, 
even  though  the  trajectories  were  different. 

[Lev95,  HNP97]  also  cited  similar  software  safety  issues  as  the  cause  of  the  Therac- 
25  computer-controlled  radiation  therapy  machine  accidents,  which  caused  massive  radia¬ 
tion  overdoses  in  six  people  between  June  1985  and  January  1987.  [WLL+01]  introduced  a 
hierarchical  accident  model  to  help  identify  safety  factors  considerations  with  three  levels 
of  abstraction:  1)  Level  I  factors  include  the  chain  of  events;  2)  Level  II  factors  included 
the  conditions  or  lack  of  conditions  allowing  the  events  at  Level  I  to  occur;  and  3)  Level  III 
factors  include  the  root  cause  or  systemic  factors  of  an  accident. 

Satisfaction,  the  final  [ISOOl]  quality-in-use  characteristic  includes  the  software’s 
ability  to  satisfy  users.  There  are  also  a  myriad  of  issues  the  Department  must  address  to 
achieve  customer  satisfaction  and  confidence  including: 

•  Credibility  in  Department  M&S, 

•  Improved  Department  decision-makers  confidence  in  simulation  results, 

•  A  quantifiable  return-on-investment  (ROI)  methodology  for  simulations, 

•  Additional  validated  automated  processes  and  tools  for  models  and  simulations, 

•  Simulation  utility  keeping  pace  with  user  requirements  and  decision-makers  expec¬ 
tations  [SchOlc], 

•  Interoperability  of  legacy  M&S  systems  with  war  fighting  systems  [DoDOla], 
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3.  Testing  for  Quality  Attributes 


[BM93]  identified  four  distinct  periods  of  software  development  identifiable  by  the 
following  dominant  themes  and  goals:  1)  the  functional  era  (1960s),  2)  the  schedule  era 
(1970s),  3)  the  cost  era  (1980s),  and  most  recently,  4)  the  quality  era  (1990s).  Software 
testing  [DPR+94,  Roa98,  ClaOl,  PLP01,  RakOl]  is  a  critical  aspect  of  quality  analysis  and 
is  most  effective  when  based  on  quantitative  measures  integrated  in  the  development  envi¬ 
ronment,  and  supporting  the  test  management  process.  Software  testing  is  a  complex  un¬ 
dertaking,  which  addresses  many  dimensions  including  new  test  challenges  for  compo¬ 
nents,  Java,  active  agents,  object-oriented  technology,  software  reliability  engineering 
[IC99,  ICO  1],  website  testing,  test  management,  and  the  human  element.  Additional  soft¬ 
ware  test  activities  and  challenges  include  the  employment  of  Bayesian  graphical  models 
[WGC02],  the  context  of  software  testing  in  the  development  process,  the  different  roles  of 
testers  and  developers  [WhiOO],  statistical  software  engineering  [JA97],  and  test  case  pri¬ 
oritization  to  increase  the  rate  of  fault  detection  [EMR02], 

Software  testing  is  an  essential  component  of  simulation  software  development,  a 
critical  technique  for  evaluating  product  quality,  and  essential  for  improving  product  qual¬ 
ity  by  identifying  defects  and  problems  early  in  the  development  process  [Som95,  Pre97, 
Roa98,  RakOl].  Software  companies  face  major  challenges  testing  products  and  [WhiOO] 
predict  these  challenges  will  continue  to  grow  with  the  increasing  complexity  levels  of  the 
software.  [RTI02]  cited  improved  software  testing151  as  a  critical  factor  supporting  soft¬ 
ware  quality  and  reducing  the  software  errors,  estimated  to  cost  the  U.S  economy  $59.5 
billion  annually. 

After  recent  mission  failures,  NASA  developed  an  independent  verification  and 
validation  (IV&V)  implementation  approach  [RosOl]  for  all  agency  software  development 
with  a  focus  on  high-risk  cutting  edge  projects.  NASA  based  the  new  IV&V  approach  on 
nine  factors,  which  NASA  determined  impacted  software  development  including:  software 
team  complexity,  contractor  support,  organization  complexity,  schedule  pressure,  process 


Software  testing  is  the  process  of  applying  metrics  to  determine  product  quality  and  is  the  dynamic  execution  of  software  and  the 
comparison  of  the  results  against  a  set  of  pre-determined  criteria.  [RTI02] 
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maturity  of  software  provider,  degree  of  innovation,  level  of  integration,  requirements  ma¬ 
turity,  and  software  lines  of  code,  evaluated  against  five  risk  categories  [RosOl]. 

Software  testing  is  also  a  best  practice  process  and  technique  in  the  Department’s 
developmental  test  and  evaluation  procedures  according  to  [SNF+99].  Although  they  share 
similar  test  techniques  and  methods,  [BCC+Olb]  differentiates  software  testing  from  soft¬ 
ware  maintenance,  noting  that  software  testing  is  a  component  of  the  development  process, 
whereas,  software  maintenance  addresses  system  failures  after  the  software  is  delivered. 
Software  testing  may  also  include  static  verification  techniques.  Software  testers  normally 
conduct  testing  at  the  unit,  integration,  and  system  level,  applying  a  number  of  techniques 
including  black-box  and  white-box  testing,  supporting  different  objectives  cited  by  [Roa98, 
RakOl]: 

•  Acceptance  /  Qualification  testing, 

•  Installation  testing, 

•  Alpha  and  Beta  testing, 

•  Conformance  /  Functional  /  Correctness  testing, 

•  Reliability  /  Achievement  /  Evaluation  testing, 

•  Regression  testing, 

•  Performance  testing, 

•  Stress  testing, 

•  Back-to-back  testing, 

•  Recovery  testing, 

•  Configuration  testing, 

•  Usability  tests  [Roa98,  RakOl]. 

New  verification  techniques  continue  to  evolve  and  now  include  the  time  warp 
technique  for  synchronizing  parallel  discrete  event  simulators  [FRC+02];  model-based 
verification  [GB99];  model  extraction  techniques  for  automated  verification  [HS02];  assur¬ 
ance-based  testing  [Pau99];  compositional  verification  methods  [WH02];  and  monitors 
employed  as  oracles  or  supervisors  to  analyze  target  systems  [PP02].  [DMSOOb]  proposed 
that  component-based  M&S  methodology  articulated  by  [HusOO]  “could  allow  more  effec¬ 
tive  and  affordable  M&S  verification”  [HusOO] . 
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4.  Model  and  Simulation  Quality  Attributes 

The  M&S  literature  contains  a  wide  range  of  M&S  quality  dimensions  and  attrib¬ 
utes  supporting  key  M&S  credibility  concepts,  including  the  terms  fidelity  and  accuracy. 

In  practice  today,  the  Department’s  M&S  community  often  use  the  terms  fidelity  and  accu¬ 
racy  synonymously.  Unfortunately,  the  Department’s  M&S  community  lacks  consensus  on 
a  fidelity  definition,  how  to  measure  it,  what  it  costs,  or  even  its  relative  importance  for 
achieving  confidence  and  credibility  in  M&S  results.  The  casual  use  of  other  terms  closely 
related  to  fidelity  such  as  resolution,  detail,  aggregation,  and  multi-resolution  add  to  the 
general  confusion  limiting  the  common  understanding  of  fidelity.  This  adversely  affects 
simulation  credibility  and  confidence  in  the  simulation  results. 

a.  Fidelity 

Although  fidelity  is  a  critical  component  for  credible  M&S,  it  has  proven 
elusive  to  implement  and  resisted  multiple  theoretical  efforts  [Bai92,  Ham96,  GF97,  FY97, 
Pac97,  SFL+97,  HNP97,  Fay98a,  Fay98b,  GFT98,  GZ98,  Had98,  Har98b,  McD98, 

Pac98a,  Pac98b,  RGJ98,  SBL98,  Pea99,  Har99b,  MBH99,  RVJ+99,  GJOO,  LMR+00, 
PacOlb,  HYOla,  HYOlb,  BP02,  RIO02]  to  describe  in  objective  or  quantitative  terms.  In 
addition,  [HNP97]  suggest  bounding  fidelity,  defining  perfect  fidelity  as  a  simulation  indis¬ 
tinguishable  from  reality,  which  may  be  possible  for  some  systems  (e.g.,  virtual  reality), 
although  still  out  of  the  theoretical  realm  for  non- trivial  simulations. 

The  [DoD95]  addressed  the  issues  of  fidelity  in  three  of  the  six  major  objec¬ 
tives  (e.g.,  objectives  two,  three,  and  four).  The  ability  to  describe  and  quantify  M&S  fi¬ 
delity  or  “goodness”  appropriately  is  essential  for  fully  achieving  other  objectives  identi¬ 
fied  in  the  master  plan,  such  as  HLA  (Sub-Objective  1-1),  CMMS  (Sub-Objective  1-2),  and 
support  of  endeavors  such  as  Simulation  Based  Training,  Analysis,  and  Acquisition  (Sub- 
Objective  5-1).  Systemic  issues  confronting  improved  fidelity  include  challenges  with 
physical,  visual,  audio,  motion,  and  temporal,  environmental  and  behavioral  fidelity 
[DoD95], 
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The  [DoD95,  DoD98]  references  contain  the  Department’s  approved  defini- 
tion  for  fidelity  .  According  to  the  [RPGOO],  fidelity  is  a  core  concept  involved  in  every 
issue  in  M&S,  especially  issues  related  to  V&V.  Fidelity  is  also  one  of  the  key  areas  di¬ 
rectly  affecting  the  credibility  of  the  Department’s  M&S  programs  and  the  confidence  in 
the  simulation  or  federation  results  provided  to  decision-makers.  Fidelity  is  inherently  an 
abstraction  or  a  representation  of  some  component  of  reality,  a  simuland  ,  or  referent  de¬ 
veloped  in  M&S.  Table  6-2  provides  an  overview  of  the  fidelity  context  from  the 
[RPGOO]. 

However,  in  practice,  the  M&S  community  describes  fidelity  by  a  number 
of  dimensions  and  attributes:  -  accuracy,  resolution,  aggregation,  de-aggregation,  detail, 
extent,  granularity,  precision,  repeatability,  time,  and  spatial  consistency.  Many  of  these 
tenns  are  ill  defined  and  often  cited  interchangeably  in  the  current  literature.  [SFL+97]  de¬ 
fine  “fidelity”  as  a  measure  of  how  accurately  and  realistically  a  simulator  or  simulation 
represents  the  “real”  world.  [GFT98]  provide  a  definition  for  fidelity  noting  “the  extent  to 
which  the  model  reproduces  the  referent,  along  one  or  more  aspects  of  interests”  [GFT98]. 
[RGHOO]  provide  two  additional  definitions  of  fidelity  with  the  caveat  that  fidelity  may  de¬ 
scribe  model  representations,  a  simulation,  simulation  data,  or  an  exercise,  with  different 
implications  for  each  use.  [Ham96,  HNP97]  addressed  the  benefits  of  model  simplifica¬ 
tion1^4  and  decomposition155,  without  a  loss  of  fidelity,  especially  of  non-trivial  systems. 

There  are  many  more  definitions  of  fidelity  in  common  use  today,  exacer¬ 
bating  the  issue  of  defining  fidelity.  By  1992  there  were  at  least  twenty-two  different  defi¬ 
nitions  of  the  tenn  fidelity  cited  by  [LA92]  for  distributed  interactive  simulations,  further 
subdivided  into  twenty  different  hardware  and  software  components,  requiring  different 
levels  of  fidelity.  The  heuristic  practice  for  describing  fidelity  includes  short  imprecise, 
subjective,  qualitative  descriptions  for  fidelity  such  as  low,  medium  or  high.  In  addition  to 
these  “short  descriptions”  for  fidelity,  [RGHOO]  identifies  “shorthand  descriptions”  such  as 
the  Level  D  flight  simulation  classification  consisting  of  over  100  attributes  used  by  the 
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Fidelity  is  specified  as  the  accuracy  of  the  representation  when  compared  to  the  real  world  [DoD95,  DoD98] 

153  .  .  ..... 

The  simuland  is  the  representation  the  simulation  is  intended  to  represent,  not  necessarily  the  real  world,  with  the  referent  being  the 

sum  total  of  what  is  known,  assumed,  or  projected  about  the  simuland  [RPGOO]. 

154 

Model  simplification  includes  three  classes:  brevity,  clarity,  and  efficiency  [HNP97]. 

An  alternative  strategy  to  simplification,  decomposition  breaks  the  system  down  into  component  parts  to  model  [HNP97]. 
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Federal  Aviation  Administration,  and  “long  descriptions”  of  multiple  attributes  involving 
enumeration,  which  requires  quality  attributes  such  as  accuracy,  resolution,  or  both. 

[Gro99,  RGHOO]  characterizes  past  approaches  to  specifying  or  measuring  fidelity  as  ad 
hoc,  artistic,  or  problem  /  domain  specific. 

[GFT98]  propose  that  fidelity  is  the  key  to  simulation  validation,  however, 
[Gro99]  states,  “fidelity  has  to  be  the  least  consistently  used,  yet  most  commonly  used  tenn 
in  the  M&S  community”  [Gro99].  [Mey98]  concurs  with  [Gro99]  and  considers  the  tenn 
fidelity  overloaded,  with  no  context-free  meaning,  and  almost  useless  in  addressing  the 
goodness  of  a  simulation  or  federation.  [PacOlb]  adds  that  no  widely  accepted  set  of  terms 
yet  exists  for  the  definitions  of  uncertainty,  error,  and  variability  as  they  apply  to  M&S  fi¬ 
delity,  validity,  and  their  referents. 

[GF97]  proposed  that:  1)  fidelity  requires  different  measurements  since  there  are 
fundamentally  different  aspects  of  fidelity;  2)  fidelity  is  a  property  of  simulation  models; 
and  3)  fidelity  exhibits  certain  properties.  Later,  [GFT98,  Pac98b]  suggested  the  way  to 
improve  decision-makers  credibility  in  the  simulation  and  confidence  in  the  simulation  re¬ 
sults  is  to  describe  fidelity  and  accuracy  quantitatively.  However,  [Gro99,  RGHOO]  con¬ 
tends  there  are  two  major  obstacles  blocking  the  development  of  a  fidelity  measurement 
standard:  1)  there  must  exist  an  accepted  definition  of  the  real  or  imagined  world  with  suf¬ 
ficient  quantifiable  characteristics  to  measure  the  difference  between  it  and  the  simulation; 
and  2)  the  simulation  must  be  similarly  defined. 

Fidelity  according  to  [Mey98]  addresses  the  M&S  measure  of  agreement 
with  the  perceived  reality  within  a  specific  context  and  suggests  that  fidelity  is  relative  to 
simulation  resolution,  appropriate  for  detennining  differences  between  the  resolution  of  the 
simulation  and  the  details  and  accuracy  of  the  models.  A  major  cooperative  govern¬ 
ment/industry/academia  consortium  collaborated  on  DIS  interoperability  standards  for 
IEEE  approval  from  1989-1995.  The  objective  of  the  collaborative  initiative  P1278.5  enti¬ 
tled  Standard  for  Distributed  Interactive  Simulation-Fidelity  Description  Requirements 
[IEE95,  STR96]  was  the  interconnection  of  dissimilar  components  of  DIS  to  provide  real¬ 
time  applications  with  appropriate  fidelity.  Shelved  in  1995,  the  [IEE95]  failed  to  span  the 
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problem  space,  generated  insufficient  support  competing  against  the  emerging  HLA  stan¬ 
dard,  and  lacked  community  support  for  its  approach  to  fidelity. 

The  simulation  literature  has  a  deep,  diverse  body  of  theory  on  fidelity  and 
its  impact  on  M&S  credibility.  [Pac98]  reviewed  basic  fidelity  concepts,  definitions  and 
theory,  noting  the  need  to  be  concerned  about  M&S  fidelity.  [Pac98]  also  provides  a  syn¬ 
opsis  of  fidelity  ideas  and  issues  providing  a  framework  for  dealing  with  fidelity  through¬ 
out  the  M&S  development  life  cycle.  In  addition,  [Pac98]  suggests  methods  to  measure, 
estimate  or  quantify  fidelity  including  adopting  a  methodology  for  deriving  and  publishing 
M&S  fidelity  metrics;  economic  issues;  and  other  challenges  which  have  plagued  analytical 
approaches,  including  classical  and  Bayesian  statistics,  game  and  decision  theory.  How¬ 
ever,  [Mey98]  provides  a  counterpoint,  arguing  that  quantifying  M&S  fidelity  would  be 
extremely  questionable  since  M&S  fidelity  has  almost  no  context-free  definition. 

[Hug97b]  further  proposes  that  new  technologies  support  1)  increased  physical  fidelity,  2) 
improved  reasoning  fidelity,  3)  enhanced  behavioral  fidelity,  4)  better  flexibility,  5)  im¬ 
proved  infonnation  transfer  and  fusion. 

The  failure  to  describe  M&S  fidelity  appropriately  may  adversely  influence 
current  and  future  Department  interoperability,  transformation,  business,  and  warfighting 
initiatives.  [PacOlb]  proposes  a  referent156  for  fidelity  and  validity,  where  the  fidelity  ref¬ 
erent  provides  the  system  theory,  and  conceptually  is  the  most  complete  collection  of  in¬ 
fonnation  about  the  subject  represented  in  the  simulation,  while  the  validity  referent  is 
more  complex  because  it  must  involve  intended  use  for  the  simulation.  The  referent  is  a 
standard  from  which  the  thing  being  represented  in  M&S  is  derived  or  the  standard  against 
which  the  correctness  of  M&S  representation  is  measured  [PacOlb].  [PacOOa]  also  dis¬ 
cusses  approaches  for  a  conceptual  model,  supporting  enhanced  model  completeness,  con¬ 
sistency,  and  coherency. 

[Had98]  considers  fidelity  as  an  open  issue,  with  respect  to  fidelity  and  reso¬ 
lution,  scope,  and  completeness  from  the  aspect  of  M&S  developer.  Moreover,  [GF97]  and 
[GFT98]  note  that  M&S  fidelity  is  a  critical  measure  of  M&S  credibility,  and  one  the  larg¬ 
est  cost  drivers  for  M&S,  although  [GFT98]  believes  that  fidelity  is  currently  of  little  use 

156  A  referent  is  a  codified  body  of  knowledge  about  a  thing  being  simulated  [PacOlb], 

148 


when  engineering  the  M&S  requirements.  In  addition,  [Pac97,  Pac98a]  notes  that  there  is 
no  specific  guidance  for  fidelity  and  that  the  M&S  community  has  not  yet  agreed  upon  ba¬ 
sic  fidelity  concepts  and  definitions.  [PacOlb]  also  provides  a  broad  overview  of  the  issues 
confronting  improved  simulation  and  federation  credibility  and  confidence  in  the  results, 
including  the  important  topic  of  quantifying  the  validity  of  HLA  federates  and  federations. 

In  addition,  [Had98]  describes  the  problem  of  defining  an  appropriate  level 
of  fidelity  for  Conceptual  Models  of  the  Mission  Space,  and  notes  fidelity  is  the  most  diffi¬ 
cult  characteristic  to  define  because  it  applies  to  many  aspects  of  a  problem.  For  example, 
a  simulation  may  contain  a  “high  fidelity”  with  respect  to  a  sensor  representation,  but  still 
labeled  as  “low  fidelity”  with  respect  to  other  representations  such  as  human  behavior  or 
threat.  [RGHOO]  added  that  fidelity  requirements,  specifications,  and  conceptual  models 
have  different  research  approaches  although  they  share  many  shortcomings,  including  the 
lack  of  a  formal  and  fundamental  theory  or  framework.  The  [RPGOO]  guide  provides  a  fi¬ 
delity  conceptual  framework  in  Figure  6-2  with  fidelity  as  a  core  concept  integral  to  all  as¬ 
pects  of  M&S,  especially  VV&A;  comprised  of  attributes  such  as  resolution,  er- 
ror/accuracy,  sensitivity  ,  and  precision;  and  concluding  that  qualitative  terms  inade¬ 
quately  express  fidelity. 

During  the  decade  of  the  1990’s,  the  Department  and  M&S  community  in¬ 
vested  significant  time,  resources  and  attention  to  improve  M&S  verification,  validation, 
and  accreditation  (VV&A).  These  VV&A  efforts  established  processes  and  identified 
VV&A  responsibilities,  but  have  not  provided  methods  for  quantifying  fidelity  require¬ 
ments  or  for  determining  complex  M&S  accuracy  attributes  quantitatively.  Other  groups, 
especially  those  involved  in  the  Department  of  Energy’s  Advanced  Strategic  Computing 
Initiative  (ASCI)  and  others  concerned  with  M&S  of  basic  physical  processes,  have  started 
significant  efforts  to  quantify  M&S  validity  [PacOlb],  These  groups  have  begun  to  address 
questions  of  uncertainty  and  error  for  both  M&S  and  experimental  data  used  with  the  M&S 
as  inputs  and  as  standards  for  validity  comparisons. 
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Sensitivity:  The  ability  of  a  component,  model  or  simulation  to  respond  to  a  low  level  stimulus  [RPGOO]. 
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Figure  6-2.  Overview  of  Fidelity  Context  (From  [RPGOO]) 
b.  Accuracy 

Accuracy,  a  sub-characteristic  of  the  [ISOOl]  quality  characteristic  dis¬ 
cussed  earlier  also  plays  a  major  role  as  an  M&S  quality  attribute.  Department  M&S  users 
also  demand  more  accurate158  elements  of  the  mission  space  and  developing  authoritative 
representations  remains  a  major  objective  of  the  draft  Department  M&S  Master  Plan, 
[DoDOla]  currently  under  development.  This  has  proved  challenging  to  the  Department 
since  there  are  many  views  of  accuracy.  For  instance,  [Mey98]  excludes  the  term  simula¬ 
tion  in  his  definition  and  proposes  that  accuracy  “is  a  measure  of  the  exactness  of  the 
model  with  respect  to  the  characteristics  and  behaviors  of  the  physical  entity  which  the 
model  represents”  [Mey98]. 


158  Accuracy  is  1)  the  degree  of  exactness  of  a  model  or  simulation,  high  accuracy  implying  low  error.  Accuracy  equates  to  the  quality 
of  the  result,  and  is  distinguished  from  precision ,  which  relates  to  the  quality  of  the  operation  by  which  the  result  is  obtained  and  may  be 
repeated  [DoD98],  2)  the  degree  to  which  a  parameter  or  variable  or  set  of  parameters  or  variables  within  a  model  or  simulation  conform 
exactly  to  reality  to  reality  or  to  some  chosen  standard  or  referent  (SISO)  [RPGOO]. 
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Providing  another  viewpoint,  [Bal98]  addressed  the  significant  accuracy 
quality  characteristics  inherent  in  verification,  validation,  testing,  accreditation,  certifica¬ 
tion,  and  credibility  assessment  activities.  In  yet  another  view  for  the  perspective  of  the 
data,  [LevOO,  Kil02]  propose: 

•  That  data  accuracy  requires  the  appropriate  data  with  the  correct  resolution  for  the 
intended  use, 

•  The  proper  quality  of  data  established  by  the  data  producer  and  reviewed  by  the 
user, 

•  The  correct  data  transformation  activities  to  make  the  data  compatible  with  M&S 
[LevOO,  Kil02]. 

There  are  also  many  other  views  of  accuracy.  [PacOOa]  addresses  conceptual  model 
decomposition,  and  how  the  characteristics  of  the  simulation  elements  are  abstracted  de¬ 
termine  the  accuracy  and  precision.  [FY97]  defines  accuracy  as  being  inversely  propor¬ 
tional  to  an  error  measurement  in  which  more  accuracy  implies  less  error.  [FY97]  also  dis¬ 
cusses  how  to  detennine  and  trace  M&S  accuracy  through  the  models  and  processes  asso¬ 
ciated  with  each  layer,  ensuring  that  the  accuracy  meets  the  design  requirements,  ensuring 
the  M&S  operates  within  the  maximum  error  bounds  and  thus  meet  its  intended  require¬ 
ments. 

[Gro99,  RGHOO]  suggest  accuracy  is  determined  by  how  well  the  simulation  algo¬ 
rithm  represents  the  subject  that  is  simulated,  and  may  be  measured  against  reality,  termed 
real  accuracy159;  or  against  the  articulation  of  the  an  application  domain  or  mission  space; 
abstraction  accuracy160;  and  should  to  relate  to  a  single  parameter  or  set  of  parameters. 

With  the  intent  of  achieving  software  engineering  consistency,  in  this  research,  we  employ 
the  IEEE  definition  for  accuracy161  provided  in  the  HLA  Framework  and  Rides  [IEEOOc]. 


159 

Real  accuracy  is  a  function  of  both  the  correctness  of  the  abstraction  and  of  the  simulation’s  representation  [Gro99,  RGHOO]. 

160  Abstraction  accuracy  refers  to  the  only  the  function  of  the  simulation’s  representation  of  the  abstraction  [Gro99,  RGHOO]. 

161  Accuracy  is  the  measure  of  the  maximum  deviation  of  an  attribute  or  parameter  value  in  simulation  or  federation  from  reality  or  some 
other  chosen  standard  or  referent  [IEEOOc]. 
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c. 


Precision 


Precision  is  also  a  close  relative  to  accuracy  and  often  used  as  a  synonym  in 
the  body  of  research.  [Gro99,  RGHOO]  suggests  that  precision162  is  a  measure  of  the  reso¬ 
lution  or  granularity  with  which  a  parameter  may  be  determined,  and  always  limits  accu¬ 
racy163.  Precision164  receives  additional  clarification  in  the  [RPGOO].  Preferring  quantita¬ 
tive  attributes  to  qualitative  identifiers,  the  selected  IEEE  standard  glossary  of  software  en¬ 
gineering  terms  [IEE90]  definition  for  precision165  suggests  improved  correctness  in 
verification  and  validation  processes,  inferring  higher  credibility  in  the  simulation  and 
improved  confidence  in  the  results. 

d.  Timeliness 

Timeliness166,  a  factor  in  considering  simulation  fidelity,  cautions  [Gro99, 
RGHOO]  has  the  potential  to  create  issues  for  parameter  accuracy  and  precision  in  the  parts 
of  the  simulation  or  federations,  which  may  advance  faster  or  slower  than  other  parts. 
[Gro99,  RGHOO]  indicates  these  manifestations  may  occur  in  distributed  simulations,  in 
unitary  simulations  employing  distributed  processing  techniques,  or  in  discrete  event  simu- 
lations,  and  is  dependent  on  time  management  techniques  in  the  simulation  . 

e.  Error 
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Error  is  a  significant  M&S  quality  attribute.  [Ham96,  HNP97]  note  an  er¬ 
ror  occurs  when  a  fault  (e.g.,  a  physical  defect  or  flaw  occurring  in  either  the  hardware  or 


162  Precision  may  normally  be  determined  in  a  simulation  by  an  understanding  of  the  systems  computational  processes  such  as  the  num¬ 
ber  of  significant  digits,  round  off,  interpolation  intervals,  update/reffesh  rates,  and  the  quality  of  the  real  world  data  [Gro99,  RGHOO]. 

Accuracy,  precision,  and  timeliness  characteristics  describe  how  close  the  representation  of  an  individual  parameter  is  to  reality 
[Gro99,  RGHOO]. 
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The  SISO  further  defines  precision  as  a  1.  The  quality  or  state  of  being  clearly  depicted,  definite,  measured  or  calculated,  2.  A  qual¬ 
ity  associated  with  the  spread  of  data  obtained  in  repetitions  of  an  experiment  as  measured  by  variance,  the  higher  the  precision,  3)  A 
measure  of  how  meticulously  or  rigorously  computational  processes  are  described  or  performed  by  a  model  or  M&S  (SISO)  [RPGOO]. 

Precision  is  the  degree  of  exactness  or  discrimination  with  which  quantity  is  stated  (e.g.,  precision  of  3  decimal  places  versus  a  6 
decimal  palace  precision)  [IEE90]. 

Timeliness  is  impacted  by  the  maximum  magnitude  of  error  between  the  parameter  with  and  without  the  missed  updates,  caused  by 
communications  delays  or  an  excessive  computation  time,  supports  quantifying  the  impact  of  timeliness  on  a  specific  parameter  [Gro99, 
RGHOO]. 

167  Time  in  a  simulation  may  be  managed  with  various  techniques  such  as  continuous  time,  time  step,  discrete  event,  etc  [Gro99, 
RGHOO,  RGHOO]. 

Error  is  the  difference  between  a  calculated,  measured,  or  observed  value  and  a  correct  value  (SISO)  [RPGOO]. 
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software)  becomes  apparent,  adding  that  if  the  error  causes  the  system  to  perform  a  func¬ 
tion  incorrectly,  a  failure  occurs.  According  to  [Gro99,  RGHOO],  any  review  of  simulation 
fidelity  involves  factoring  in  error.  Possible  M&S  error  sources  include  imperfect  meas¬ 
urement  of  input  data,  deviation  from  correct  data,  imperfect  algorithms,  and  the  finite 
limitations  of  logical  and  computational  processes. 

A  report  on  software  error  analysis  [PW93]  sponsored  by  the  NIST  found  a 
wide  variation  of  error  techniques  for  high  integrity  software  and  proposed  the  develop¬ 
ment  of  an  organizational  error  analysis  database  identifying  the  most  effective  techniques 
to  support  error  detection.  Moreover,  the  impact  of  error  in  software-intensive  systems, 
including  simulation  software  is  staggering.  Of  the  $59.5  billion  lost  annually  to  the  U.S. 
economy  do  to  prevalent  software  errors;  half  of  the  cost  is  borne  by  the  develop¬ 
ers/vendors,  with  the  other  half  of  the  costs  paid  by  the  users  [RTI02].  Over  half  of  these 
errors  were  uncovered  late  in  the  development  process  or  while  in  use  after  the  sale 
[RTI02], 


f.  Resolution 

A  significant  M&S  community  issue  includes  the  development  of  commu¬ 
nity  standards  defining  resolution  of  model  representations  for  use  throughout  the  life  cycle 
of  the  systems.  [DoD98]  defines  resolution169,  while  [RGHOO]  view  resolution  as  “the  ex¬ 
tent  to  which  the  M&S  models  each  aspect  of  the  real  world”.  A  definition  in  the  web- 
based  [RPGOO]  expanded  the  meaning  of  resolution  .  The  [RPGOO]  definition  also  ad¬ 
dressed  granularity  or  the  reduction  of  something  into  related  parts  or  components. 

[HNP97]  propose  that  the  question  of  detail  affects  simulation  resolution,  with  both  factors 
related  to  fidelity.  [Mey98]  finds  resolution  erroneously  used  to  describe  the  result  of  some 
measurement,  rather  than  correctly  identifying  the  resolution  of  the  device  taking  the  meas¬ 
urement  (e.g.,  a  sensor). 

[Mey98]  also  proposes  that  M&S  resolution  like  fidelity,  derives  from  the 
context  of  a  simulation,  and  is  a  measure  of  the  minimum  degree  that  the  constituent  mod- 

169 

Resolution  is  the  degree  of  detail  and  precision  used  in  the  real  world  aspects  in  a  model  or  simulation  [DoD98]. 

170 

The  [RPGOO]  defines  resolution  as  1 )  the  degree  of  detail  used  to  represent  aspects  of  the  real  world,  or  a  specified  standard  or  refer¬ 
ent  by  a  model  or  simulation,  2)  separation  or  reduction  of  something  into  its  constituent  parts;  granularity  (SISO)  [RPGOO]. 
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els’  accuracy  and  detail  correspond.  The  semantic  relationship  between  the  terms  pre¬ 
sented  in  Figure  6-2  delineates  the  delta  between  the  fidelity  required  by  the  application  or 
tolerances  that  define  the  range  of  values  for  dependent  and  independent  variables,  and 
the  fidelity  present  in  a  model  in  a  simulation,  the  knowable  quantity.  Figure  6-2  also  illus¬ 
trates  that  physical  reality,  either  material  or  imagined,  provides  the  basis  for  obtaining  all 
knowledge  of  reality. 

Known  reality  manifests  this  body  of  knowledge.  Known  reality  also  pro¬ 
vides  the  source  for  referents  supporting  application  requirements  and  model  or  simulation 
fidelity,  and  for  abstractions  of  reality,  in  models  and  simulations.  [HNP97]  suggest  that 
model  resolution,  a  critical  component  when  determining  the  usefulness  of  a  simulation  is 
conceptually  closely  related  to  fidelity,  however,  caution  that  one  does  not  imply  the  other, 
and  higher  resolution  does  not  necessarily  increase  fidelity.  However,  since  resolution  af¬ 
fects  federation  credibility  supporting  confidence  in  the  results,  the  [IEEOOc]  definition  for 
resolution  is  the  most  appropriate. 

g.  Detail 

Simulation  detail,  like  resolution,  also  includes  many  facets  and  views. 
[RR83]  state  that  the  model  level  of  detail  describes  the  number  of  functions,  or  level  of 
structure  included  in  the  model,  while  the  extent,  a  similar  concept,  relates  to  the  range  of 
system  functions  within  the  simulation.  However,  [Mey98]  defines  detail  as  a  measure  of 
model  complexity  and  completeness  when  compared  to  the  characteristics  of  the  physical 
entity  represented  by  the  model. 

[Fay98a,  Fay98b]  proposes  a  method  for  measuring  the  level  of  detail  in  a 
simulation  based  on  identifying  the  important  parameters  and  a  sensitivity  analysis  ap¬ 
proach,  which  provides  a  categorized  hierarchy  of  phenomena,  model,  and  value  parame¬ 
ters.  [Had98]  further  suggests  that  the  sophistication  of  the  developers  in  the  problem  do- 


Tolerance:  1 .  The  maximum  permissible  error  or  the  difference  between  the  maximum  and  minimum  allowable  values  in  the  proper¬ 
ties  of  any  component,  device,  model,  simulation,  or  system  relative  to  a  standard  or  referent.  Tolerance  may  be  expressed  as  a  percent 
of  nominal  value,  plus  and  minus  so  many  units  of  a  measurement,  or  parts  per  million.  2.  The  character,  state  or  quality  of  not  interfer¬ 
ing  with  some  thing  or  action  [RPGOO]. 

172 

The  IEEE  HLA  definition  of  resolution  describes  it  as  the  smallest  resolvable  value  separating  attributes  or  parameter  valuses  that 
can  be  discriminated.  Resolution  may  vary  with  magnitude  for  certain  data  types  [IEEOOc]. 
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main  factors  significantly  in  the  development  of  the  appropriate  level  of  detail  in  the  appli¬ 
cation. 


h.  Aggregation  /Disaggregation 

Aggregating  and  disaggregating  model  representations  present  significant 
challenges  to  the  development  of  credible  simulations  [HNP97].  The  many  complex  issues 
involving  aggregation  /  disaggregation  complicate  the  development  of  improved  M&S 
credibility.  The  [DoD02d]  notes  the  lack  of  predominate  type  of  M&S  and  identifies  the 
need  for  varying  levels  of  aggregation  and  flexible  interoperability  standards  to  meet  di¬ 
verse  customer  needs.  Seeking  more  definitive  answers,  [HJ95]  explored  several  different 
approaches  to  aggregation,  and  established  requirements  to  develop  consistency  between 
aggregated  and  higher  fidelity  representations,  concluding  that  common  intuition  about 
cause,  effects  and  outcomes  are  often  incorrect. 

[BidOO]  addressed  additional  areas  for  representing  battlespace  objects  as 
elements  of  other  aggregate  element  representations,  including  mechanisms  for  establishing 
aggregated  representations  on  a  permanent  or  temporary  basis  .  Taking  an  object  ori¬ 
ented  approach,  [HHPOO]  proposed  a  process  transitioning  aggregated  /  decomposed  mod¬ 
els  into  an  object-oriented  representation  by  defining  modules  in  an  object  oriented  domain. 
The  object  oriented  methodology  supports  other  possible  structures  proposed  by  [Tra93, 
CRB01,  CM02], 

The  Department  also  requires  credible,  authoritative  representations  includ¬ 
ing  aggregated  and  disaggregated  system  representations  of  U.S.,  allied,  friendly,  paramili¬ 
tary,  coalition,  neutral,  threat,  systems,  and  for  combat,  combat  support  and  combat  service 
support  systems  and  processes.  This  includes  Department  objectives  for  establishing  stan¬ 
dard  taxonomies  and  common  object  classes  for  systems  representations  by  FY2004 
[DoDOla].  Two  basic  approaches  have  evolved  for  dealing  with  resolution  and  fidelity  is¬ 
sues  involved  in  aggregation  /  disaggregation:  the  top-down  and  the  bottom-up  approaches. 
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Aggregation  is  defined  as  the  ability  to  group  entities  while  preserving  the  effects  of  entity  behavior  and  interaction  while  grouped 

[DoD98]. 
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Disaggregation  refers  to  the  ability  to  represent  the  behavior  of  the  aggregated  unit  in  terms  of  its  component  entities  [DoD98]. 

175 

This  scheme  deals  with  situations  where  an  object  needs  to  be  represented  both  individually  and  as  part  of  an  aggregation  at  the  same 
time.  [BidOO] 
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Other  resolution  issues  involve  the  development  of  acceptable  algorithms  for  aggregating 
representations  of  a  single  system  into  groups  of  entities  and  disaggregation  of  grouped 
representations  [You02a,  You02b]. 

5.  Model  and  Simulation  Quality  Approaches 

Department-  and  National-level  decision-maker  and  stakeholder  knowledge  and 
perceptions  of  M&S  quality  directly  impacts  credibility  in  the  simulation  or  federation,  and 
confidence  in  the  results.  [Wal94]  described  five  approaches  for  viewing  and  defining 
quality  based  on  the  perspectives  of  different  stakeholders  —  transcendental,  user-related, 
buyer-related,  production-related,  and  process-related.  Today,  the  Department  employs  the 
“transcendental”  approach  to  M&S  software  development,  attempting  to  maintain  high 
quality  without  precisely  defining  or  measuring  the  product.  In  lieu  of  precise  measure¬ 
ments  in  M&S  software,  the  M&S  community  labels  model  representations  with  subjec¬ 
tive,  qualitative  identifiers  such  as  high,  medium,  or  low  fidelity. 

The  user-related  approach  for  user  satisfaction  described  by  [Wal94]  for  the  De¬ 
partment’s  software  intensive,  simulation  development  best  fits  the  current  verification, 
validation,  and  accreditation  processes.  The  Department’s  acquisition  process  most  appro¬ 
priately  instantiates  the  buyer-related  viewpoints  in  the  cost,  schedule,  and  performance 
parameters  of  the  acquisition  process.  Metrics,  formal  methods,  and  statistics  represent  the 
Department’s  production-related  approach,  seeking  to  attain  high  quality  by  defining  spe¬ 
cific  characteristics  of  a  product  with  precise  measurements,  while  the  many  process  mod¬ 
els  (e.g.  CMM,  CMMI)  provide  the  Department  with  process-related  approach  to  prevents 
faults  and  minimizes  rework. 

The  production-related  approach  has  the  potential  to  improve  confidence  in  large 
scale  Department  M&S  software  systems.  [Boo02]  identifies  the  “use  of  certain  reasonable 
and  quantifiable  measures  of  product  and  process”  [Boo02]  to  address  the  maturity  of  soft¬ 
ware  engineering  team  that  are  indicative  of  the  product  quality.  This  dissertation  identi¬ 
fies  a  new  architecture-centered,  product  line-based  approach  to  replace  the  existing  tran¬ 
scendental  approach  to  the  Department’s  M&S  software  development  and  acquisition,  and 
addresses  the  processes  for  a  successful  implementation. 
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D.  SUMMARY 

Chapter  VI  addressed  software  and  simulation  quality  attributes  affecting  heteroge¬ 
neous  system  representation  anomalies  and  credibility  in  Department  M&S,  especially  in 
federation  interoperability.  The  chapter  reviewed  the  internal  and  external  attributes  of  the 
ISO  9126-1  Quality  model  [ISOOl],  testing  for  quality  attributes,  and  M&S  quality  attrib¬ 
utes  addressed  in  much  of  the  current  literature,  including  an  overview  of  aggregation  and 
disaggregation.  The  chapter  included  a  discussion  on  M&S  quality  approaches. 

Quality  is  a  major  component  of  M&S  credibility.  The  ISO  Quality  Model  ad¬ 
dresses  the  six  major  internal  /  external  quality  characteristics:  functionality,  reliability,  us¬ 
ability,  efficiency,  maintainability,  and  portability;  and  the  four  qualities  in  use  characteris¬ 
tics,  allowing  users  to  achieve  specified  M&S  objectives.  The  tenn  “software  quality”  re¬ 
mains  widely  used  today,  and  open  to  subjectivity,  different  views  and  various  interpreta¬ 
tions  of  the  definitions.  Software  quality  continues  to  lag  well  behind  the  computer  indus¬ 
try’s  hardware  engineering  segment  progress  on  quality,  cost,  and  performance.  While 
computer  hardware  engineering  is  well  into  the  fourth  computer  era  identified  by  [Pre97] 
and  employs  the  production  approach  to  quality  according  to  criteria  established  by 
[Wal94],  simulation  software  quality  is  best  labeled  as  transcendental.  The  chapter  also 
reviewed  model  and  simulation  quality  attributes  including  fidelity,  accuracy,  resolution, 
error  and  detail,  within  a  fidelity  context  [RPGOO] . 

This  research  found  little  evidence  that  the  M&S  software  development  community 
will  achieve  any  consistent  future  success:  1)  developing  quality  metrics  for  M&S  quality 
attributes,  and  2)  applying  them  early  enough  in  the  M&S  software  analysis  and  design 
phases  to  quantifiably  improve  M&S  software  quality  employing  current  paradigms.  In 
addition,  many  of  the  past  recommendations  to  improve  the  Department’s  M&S  portfolio 
listed  below  have  been  reiterated  and  reissued  on  repeated  occasions,  although  it  is  uncer¬ 
tain  if  more  time  and  more  resources  will  improve  the  results: 

•  Better  verification  and  validation  (V&V)  techniques  and  accreditation  processes, 

•  Better  discipline  in  the  software  development  process, 

•  Improved  use  of  data, 

•  Improved  quality, 

•  Improved  documentation. 
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VII.  SOFTWARE  ENGINEERING  EFFORTS  SUPPORTING  CREDIBILITY 


A.  INTRODUCTION 

Chapter  VII  reviews  three  selected  areas  influencing  credibility  of  large-scale,  leg¬ 
acy  simulations  in  the  Department  including  software  process  improvement  (SPI),  the 
status  of  new  major  simulation  software  engineering  initiatives  (e.g.,  JWARS,  JSIMS, 
JMASS)  and  the  growing  software  engineering  body  of  knowledge.  The  chapter  also  in¬ 
cludes  a  discussion  of  the  challenges  generated  by  Department  contract  management  over¬ 
sight  for  achieving  quality  software  and  reducing  heterogeneous  system  representation 
anomalies,  Department  institutional  factors  affecting  M&S  credibility,  and  recent  Depart¬ 
ment  software  engineering  education  initiatives.  In  addition,  research  reveals  a  growing 
awareness  that  the  Department  needs  qualified  software  engineers  who  understand  the 
foundations  of  the  software-intensive  systems,  including  credible  M&S,  upon  which  we 
basing  our  future  security,  economic  prosperity,  and  military  preparedness. 

B.  PROCESS  IMPROVEMENT  SUPPORTING  M&S  DEVELOPMENT 

Product-based  activities  evolved  with  the  early  software  development  industry, 
supported  by  artisans  practicing  software  development  as  a  craft,  a  dependence  on  heroes, 
and  a  largely  anecdotal  body  of  knowledge.  Since  the  mid-1970s  a  view  of  building  soft¬ 
ware  as  a  process-based  activity  grew  and  began  to  challenge  the  traditional  model  of 
building  software  as  product-based  activity.  This  process-based  activity  concept  drew 
heavily  from  the  manufacturing  sector  with  a  theory  that  how  the  developer  built  the  soft¬ 
ware  affected  the  quality  of  the  end  product.  Process-based  activities  now  include  a  rapidly 
expanding  field  including  process  models,  quality  standards,  and  contractor  evaluation  cri¬ 
teria,  including  international  process  improvement,  quality  standards,  and  Department- 
based  standards. 

According  to  [SG99]  American  corporate  project  managers  for  information  tech¬ 
nology  realized  by  1996  the  true  cost,  scope,  and  size  of  software  project  failures  and  fi¬ 
nally  acknowledged  that  no  technological  or  silver  bullet  solution  existed.  The  problem 
and  solution  identified  by  [SG99]  included  people  and  processes.  [Bro96]  cited  the  need 
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for  fundamental  process  improvement  in  industry/govemment  software  development  or¬ 
ganizations  driven  by  systemic  failures  to  predict  and  meet  cost  and  schedule  targets,  a 
need  to  reduce  life  cycle  software  costs,  and  an  improved  ability  to  identify  and  address 
technical  risks. 

Process-based  activities  including  process  improvement  is  multifaceted  and  incor¬ 
porates  continuous  improvement  process  (e.g.,  Kaizen)  championed  by  Deming  [Sch91], 
business  process  reengineering  concepts  [HC93],  statistical  process  control  for  software 
improvement  [FC99,  WelOl],  and  the  process  enterprise  [HamOlb].  [BE99]  addressed 
methods  for  improving  the  process  and  product  by  optimizing  the  software  product 
throughout  the  life  cycle,  identifying  follow-on  efforts  including  product  lines  for  further 
exploration.  Process-based  activities  also  entail  event-driven  learning  approaches  for  train¬ 
ing  [HilOl],  and  project  management  processes  [Fra94],  which  provide  additional  perspec¬ 
tives  on  the  various  dimensions  of  effective  processes.  [She97]  noted  the  large  set  of  proc¬ 
ess  frameworks  caused  confusion  and  identified  six  specific  compliance  framework  catego¬ 
ries: 

•  Standards  and  Guidelines, 

•  Process  Improvement  Models  and  Internal  Appraisal  Methods,’ 

•  Contractor  Selection  Vehicles, 

•  Quality  Awards, 

•  Software  Engineering  Life-Cycle  Models, 

•  System  Engineering  Models  [She97]. 

Standards  and  guidelines  included  U.S.  military  standards  (e.g.  MIL-STD  498), 
commercial  standards  (e.g.,  Electronic  Industries  Association  (EIA)  IS  632),  and  Interna¬ 
tional  Standards  Organization  (ISO)  standards  including  the  ISO  9000  series  for  quality 
systems  and  ISO  Software  Process  Improvement  Capability  dEtermination  (SPICE),  an  in¬ 
ternational  standard  for  software  process  assessment.  Process-based  activities  generally 
defined  the  characteristics  of  good  processes,  and  did  not  prescribe  the  how  to  implement 
the  processes.  The  1991  introduction  of  the  software  Capability  Maturity  Model  (CMM) 
for  software  (SW-CMM)  [PWC+95]  built  on  product  quality  principles  promulgated  since 
the  industrial  age  production  line  practices  of  the  1930s.  The  objective  of  the  CMM  is  to 

achieve  defined,  managed,  and  optimized  processes,  with  the  expectation  that  improved 
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process  maturity  [MCR+00,  BBF+01]  will  also  yield  a  higher  quality  software  product 
[BE99], 

Follow-on  process  improvement/maturity  products  including  [Hum98a,  Hum98b, 
Hum98c]  based  on  the  same  underlying  CMM  appeared  for  other  closely  related  disci¬ 
plines  including  Software  Acquisition  [IEE98c]  CMM  [FCF+96,  FDR+98,  Gal99,  SPM99, 
GFOO],  Systems  Engineering  [H097,  Hum98a,  Hum98b,  Hum98c,  Hum99,  MCOla],  Inte¬ 
grated  Product  and  Process  Development  [Zit97],  People  CMM  [HB98c],  the  Personal 
(PSP)  /  Team  Software  Process  (TSP)  [H097,  Hum99,  VFS+99,  MCOla]  and  most  re¬ 
cently,  the  Capability  Maturity  Model  Integration  (CMMI).  In  a  similar  initiative, 
[MCR+00]  developed  a  five-stage  maturity  process  similar  to  the  CMM  for  U.S.  Govern¬ 
ment  investment  management  of  information  technology.  [WPL02]  addressed  the  possibil¬ 
ity  of  improving  M&S  credibility,  comparing  and  contrasting  CMM  key  practice  areas  with 
several  common  V&V  methods,  while  [Ric02]  discussed  the  possibility  of  the  a  M&S 
CMMI  to  better  identify  mature  companies  to  develop  and  maintain  M&S  products. 

The  Capability  Maturity  Model  for  Software  (SW-CMM)  [PWC+95]  supports  or¬ 
ganizational  development  of  improved,  common,  software  processes  based  on  a  framework 
of  best  practices,  which  impact  productivity,  performance,  cost  and  customer  satisfaction. 
There  are  five  CMM  maturity  levels  1 -Initial,  2-Repeatable,  3-Defined,  4-Managed,  and  5- 
Optimizing,  supported  by  specific  key  processes  areas  (KPA)  established  at  each  maturity 
level.  For  instance,  KPAs  for  level  2-Repeatable  include  requirements  management,  soft¬ 
ware  project  planning,  software  project  tracking  and  oversight,  software  subcontract  man¬ 
agement,  software  configuration  management,  and  software  quality  assurance. 

1  7  f. 

Acquisition  agencies  may  specify  a  SEI  Software  Capability  Evaluation  (SCE) 
method  or  Software  Development  Capability  Evaluation  (SDCE)  method  employed  by  the 
Air  Force,  to  have  a  third-party  examiner  review  a  competitors  strengths  and  weaknesses  in 
order  to  reduce  risk  to  the  acquiring  agency.  TG99]  compared  and  contrasted  both  evalua¬ 
tion  techniques  noting  both  methods  were  time  and  resource-intensive  for  both  the  contrac¬ 
tor  and  the  government.  In  addition,  [OSOO]  identified  challenges  and  inconsistencies  in 
the  SCE  process  and  suggested  improvements  to  restore  validity  to  the  process.  Confi- 

176  Software  capability  evaluations  (SCE)  are  formal,  systemic  methods  for  assessing  a  contractor’s  software  development  process 

161 


dence  in  the  evaluation  process  is  critical  since  the  most  critical  Department  acquisition 
projects,  termed  ACAT  1  programs,  must  undergo  an  SCE  or  develop  a  risk  mitigation  plan 
[Gan99], 

The  debate  over  the  value  of  software  process  improvement  continues  today.  Al¬ 
though  [FalOl]  cites  a  draft  document  defining  the  requirements  for  process  evaluation 
methods  to  improve  process  definition,  [Mos02]  contends  that  quality  of  software  process 
improvement  varies  significantly  within  the  Department  today,  and  exhibits  less  disciplined 
processes  when  compared  with  the  early  1990s.  Concerned  with  the  Department’s  soft¬ 
ware  acquisition  practices,  [AS03]  implements  Section  804  of  the  Bob  Stump  National  De¬ 
fense  Authorization  Act  for  Fiscal  Year  2003  requiring  the  establishment  of  software  ac¬ 
quisition  process  improvement  programs  by  Department  agencies  with  a  substantial  soft¬ 
ware  component,  supporting  a  Department  objective  to  improve  the  acquisition  of  soft¬ 
ware-intensive  systems. 

In  an  effort  to  improve  quality,  quality  award  initiatives  evolved  based  on  the  prem¬ 
ise  that  the  quality  of  the  processes  used  to  develop  and  maintain  a  software  system  signifi¬ 
cantly  influenced  overall  quality.  Several  organizations  developed  award  programs  to  im¬ 
prove  the  quality  situation,  including  the  Malcolm  Baldridge  National  Quality  Award,  es¬ 
tablished  by  the  U.S.  Government  in  1987,  and  the  European  Quality  Award  [She97].  The 
apparent  proliferation  of  these  models  had  several  consequences  [Cla97],  which  resulted  in 
the  initiation  of  the  CMM  Integration  (CMMI)  project  with  staged  representations 
[FAB+99a,  FAB+99b],  and  continuous  representations  [FAB+99c,  FAB+99d,  and  Ric02]. 
The  development  of  the  Capability  Maturity  Model-Integrated-Systems/Software  Engineer¬ 
ing  (CMMI-SE/SW)  built  on  three  initial  disciplines:  software  engineering,  systems  engi¬ 
neering,  and  integrated  product  and  process  development. 

Organizations  with  software  process  improvement  programs  have  reported  benefits 
in  product  quality  and  productivity,  corroborating  increased  product  quality  and  decreased 
product  costs  results  experienced  in  other  sectors  [BBF+01],  [Whi99]  addressed  the  impor¬ 
tance  of  VV&A  for  quality  software  engineering,  and  illustrated  where  VV&A  processes 
integrated  into  the  development  process  improved  cost,  performance  and  schedule.  [Bal98, 
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PW93]  also  noted  the  importance  of  establishing  an  M&S  software  quality  assurance  pro¬ 
gram,  such  as  the  program  detailed  by  [STSOO]. 

Currently,  the  software  acquisition  management  (SAM)  model  remains  a  separate 
model  awaiting  integration  into  the  current  CMMI  framework.  [San99]  identified  im¬ 
proved  process  and  product  discipline  supporting  improved  software  engineering  practices 
[DL93,  Pfl98].  Advocating  continued  professional  development,  [Whi99]  endorses  con¬ 
tinuing  training  opportunities,  and  provides  an  outline  for  training  M&S  practitioners. 
Achieving  these  objectives  requires  a  disciplined  software  engineering  approach,  mature 
software  acquisition  management  processes  [IEE98c,  SPM99,  SSJ+00],  the  capability  to 
evolve  and  improve  the  current  Department  portfolio  of  legacy  M&S,  and  maybe  most  im¬ 
portantly,  management  commitment  supporting  transfonnation  and  reengineering  initia¬ 
tives. 

Process  maturity  closely  correlates  to  improved  system  and  software  engineering 
practices  and  supports  the  management  of  large  software  projects  with  defined  consistency, 
and  process  repeatability.  [McGOl]  discusses  the  primary  benefits  and  business  case  ad¬ 
vantages  for  SPI  from  a  properly  run  SPI  program  including  improved  cycle  time,  reduced 
software  development  costs,  improved  product  quality,  reduced  maintenance  costs,  im¬ 
proved  worker  morale,  and  increased  business  competitiveness.  [McGOl]  also  identified 
possible  impacts  on  secondary  business  case  metrics  from  improved  SPI  practices  includ¬ 
ing  projected  sales,  average  historical  penalties  (e.g.,  contract  work),  personnel  turnover 
costs,  employee  development  costs,  repeat  business,  and  a  final  category,  the  risk  likeli¬ 
hood  with  and  without  software  process  improvement. 

Although  process  improvement  results  are  not  yet  conclusive,  there  are  positive  in¬ 
dications  [McGOl].  In  dealing  with  projecting  transitions  to  various  maturity  levels  a  lin¬ 
ear  correlation  was  established  between  the  Quantitative  Software  Management177  (QSM) 
quantitative  Productivity  Index  and  the  SEI  CMM  [Put92].  The  [Put92]  analysis  concluded 
that  the  number  of  companies  in  the  two  highest  maturity  levels  will  be  quite  small  for  the 
near  tenn  period,  unless  a  dramatic  change  in  the  rate  of  improvement  occurs.  In  a  follow- 


177 

QSM  has  been  in  business  capturing  management  numbers  for  software  development  activities  since  1978  and  has  a  very  large 
database  populated  with  the  complete  spectrum  of  performers  from  poor  to  very  good. 
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on  paper  [Put93]  developed  the  expected  economic  benefits  of  moving  up  the  SEI  scale.  In 
later  research,  [LFT95]  studied  the  correlation  between  the  CMM  and  software  developer 
performance,  citing  improved  cost  and  schedule  perfonnance  with  increasing  process  ma¬ 
turity  and  validating  a  correlation  between  project  success  and  CMM  ratings,  suggesting 
that  CMM  ratings  may  be  indicative  of  future  success. 

Continuing  research  by  [Cla97]  involving  one  hundred  and  twelve  projects  in  the 
sample  concluded  in  part  that  software  process  maturity  was  a  significant  factor  affecting 
software  development  effort,  and  furthermore,  that  a  one  increment  change  in  the  process 
maturity  rating,  after  nonnalizing,  resulted  in  a  “15  to  21%  reduction  in  effort”  [Cla97]. 
[Cla97]  provides  a  significant  assessment  of  the  effects  of  process  on  software  develop¬ 
ment  efforts,  and  concluded  the  CMM  is  well  defined  and  establishes  criteria  to  evaluate 
processes,  and  proposes  that  process  maturity  should  be  a  factor  in  all  cost  models. 

The  SEI  compiled  a  comparison  of  process  maturity  profiles  by  a  subset  of  the  soft¬ 
ware  community  from  1996  [SEM97]  through  1999  [SEMOO]  revealing: 

•  The  number  of  organizations  initiating  software  process  improvement  continues  to 
increase  [SEM97,  SEMOO], 

•  The  proportion  of  commercial  and  in-house  organizations  initiating  software  proc¬ 
ess  improvement  are  increasing  [SEM97,  SEMOO], 

•  Manufacturing  industries  are  conducting  the  most  software  process  assessments 
[SEM97,  SEMOO], 

•  The  service  industry  now  conducts  almost  as  many  software  process  assessments  as 
the  manufacturing  industries  [SEMOO], 

•  Nearly  half  of  the  reporting  agencies  have  less  than  100  software  personnel 
[SEM97,  SEMOO], 

•  The  overall  community  profile  continues  to  shift  towards  higher  maturity  [SEM97, 
SEMOO], 

•  The  trend  towards  higher  maturity  profile  for  offshore  organizations  compared  to 
U.S.  organizations  continues  [SEM97,  SEMOO], 

•  Software  Quality  Assurance  is  the  least  frequently  satisfied  key  process  area  (KPA) 
among  organizations  assessed  at  level  1  [SEM97,  SEMOO], 

•  Integrated  software  management,  organization  process  definition,  and  training  pro¬ 
gram  are  the  least  frequently  satisfied  level  3  KPAs  among  organizations  assessed 
at  level  2  [SEM97,  SEMOO], 

•  Higher  process  maturity  has  been  reached  among  those  organizations  reporting  re¬ 
assessments  [SEM97,  SEMOO], 
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•  All  groupings  exhibit  a  similar  pattern  for  moving  from  maturity  level  1  to  level  2 
and  from  level  2  to  level  3.  Furthermore,  level  2  to  3  tends  to  be  faster  and  have 
less  variance  [SEM97], 

•  For  organizations  that  began  their  CMM-based  SPI  effort  in  1992  or  later,  the  me¬ 
dian  time  to  move  from: 

■  Maturity  level  1  to  level  2  is  25  months, 

■  Maturity  level  2  to  level  3  is  23  months, 

■  Maturity  level  3  to  level  4  is  36  months  [SEMOO]. 

[PGWOO]  notes  the  seventy-one  level  4  or  level  5  appraised  organizations  as  of  February 
15,  2000,  continued  to  grow  since  1992  when  there  were  no  level  4  appraisals  and  only  the 
IBM  Onboard  Shuttle  team  rated  level  5  using  the  SCE  method. 

The  Department’s  draft  M&S  strategic  plan  [DoD03a]  supports  accelerated  organ¬ 
izational,  operational,  business,  and  process  reforms.  Current  challenges  to  software  proc¬ 
ess  improvement  implementation  include  the  costs  to  implement  a  SPI  program,  ensuring 
development  team  follows  processes,  establishing  process  metrics,  and  maintaining  process 
consistency.  Common  organization  barriers  to  a  SPI  program  include:  a  lack  of  resources, 
inadequate  sizing  of  the  SPI  effort,  lack  of  senior  management  sponsorship,  middle  man¬ 
agement  apathy,  and  staff  tension  [SEI95].  Building  on  the  issues  behind  poor  software 
quality  identified  by  [Dem95a]  and  [You93],  [EvaOO]  suggest  that  it  is  not  only  an  issue  of 
improving  the  software  processes,  but  also  changing  the  organization  or  project  cultures 
into  an  atmosphere  supportive  of  process  improvement. 

In  the  late  1990s,  a  countering  effort,  the  Lightweight  /  Agile  process  movement 
(e.g.,  Extreme  programming  (XP),  Adaptive  software  development)  evolved  in  a  counter¬ 
ing  effort  to  the  plethora  of  process  models  which  restricted  creativity,  according  to  the  Ag¬ 
ile  process  school  of  thought.  The  XP  School  reduces  software  development  processes  to 
the  bare  minimum  [UHC02]. 

C.  CONTRACT  MANAGEMENT 

Contract  management  oversight  [MBG+02]  is  a  critical  process  for  an  organization 
acquiring  software.  The  challenges  of  managing  the  myriad  of  contracts  supporting  the 
Department’s  IT  initiatives,  including  M&S  software  support  continue  to  grow.  Federal 
Government  spending  on  IT  services  almost  doubled  from  fiscal  years  1997  to  2001,  in- 
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creasing  from  $9  billion  to  $17  billion,  with  the  Department  of  Defense  remaining  the  larg¬ 
est  single  acquirer  of  IT  services  and  increasing  spending  by  about  forty-one  percent 
throughout  the  period  [KSZ03].  During  the  same  period  spending  on  IT  services  from  the 
General  Service  Administration  (GSA)  greatly  increased  with  spending  on  the  GSA  federal 
supply  service  schedule  growing  from  approximately  $405  million  to  $4.3  billion  [KSZ03]. 

[DoD03a]  continues  to  support  the  objective  of  increased  infonnation  sharing  and 
collaboration  within  the  Govermnent  (e.g.,  federal,  state,  local),  academia,  and  industry 
However,  the  inability  of  the  government  to  establish  M&S  as  contract  deliverables,  the 
absence  of  M&S  requirements  in  many  contract  proposal  plans,  and  the  growing  number  of 
long-term  contracts  reduce  the  potential  gains  [HBD+01].  Another  area  of  concern  in¬ 
volves  the  possible  ineffectiveness  of  the  SCE  process  [FS94]  for  contractors  identified  by 
[Mos02].  In  addition,  [HBD+01]  addressed  several  other  systemic  government-contractor 
issues  impeding  progress,  also  noted  by  other  authors  as  indicated: 

•  Collaborative  environments  misunderstood, 

•  Contractual  obligation  constraints  or  restrictions  [KM00], 

•  Only  fifty  percent  of  the  reviewed  programs  identified  M&S  in  the  prime  contractor’s 
statement  of  work, 

•  Contractor  ownership  of  reviewed  M&S  was  nearly  seventy-five  percent, 

•  Issues  with  proprietary  M&S  [Sul02],  and  a  corollary  challenge — M&S  products  (e.g., 
conceptual  models)  as  a  contact  deliverables  [Pac02d], 

•  Lack  of  management  visibility  into  programs, 

•  The  lack  of  work  breakdown  structures  [HBD+0 1  ] . 

D.  SOFTWARE  ENGINEERING  OF  MODERN  DOD  M&S 

[Ham96,  and  HNP97]  note  that  software  engineering  discipline  overlaps  and  inter¬ 
twines  with  simulation  implementation  and  the  system  modeling  process,  which  supports 
the  simulation  design.  While  reduced  costs  and  improved  performance  have  marked  the 
evolution  of  hardware  and  network  products  in  accordance  with  marketplace  competition 
and  Moore’s  Law  [Moo65],  a  recent  study  commissioned  by  the  Department  of  Com¬ 
merce’s  National  Institute  of  Standards  and  Technology  [RTI02],  concluded  that  software 
errors  are  prevalent  and  costs  the  U.S.  economy  an  estimated  $59.5  billion  dollars  annually, 
a  detrimental  figure  which  equates  to  about  0.6  percent  of  the  annual  gross  domestic  prod¬ 
uct. 
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This  is  a  systemic  Department  issue.  The  Department  created  several  organizations 
and  task  forces  over  the  past  twenty-five  years  to  evaluate  methods  for  improving  software 
quality  and  reliability,  identify  better  software  management  processes,  establish  sound 
software  engineering  practices,  reduce  software  development  cost,  and  improve  perform¬ 
ance  and  schedule  metrics  [CHK90] : 

•  In  1979  a  Joint  Logistic  Commanders  Joint  Policy  Group  on  Computer  Resources 
Management,  concerned  with  adequate  software  support  after  the  development  pe¬ 
riod,  evaluated  post-deployment  software  support  (PDSS)  procedures  supporting 
the  transition  and  operational  phases  of  the  software  system  life  cycle.  The  Depart¬ 
ment  historically  experiences  the  highest  life  cycle  costs  during  the  PDSS  of  a  soft¬ 
ware  system, 

•  The  Very-High  Speed  Integrated  Circuits  (VHSIC)  Program  created  by  the  De¬ 
partment  in  1980  supported  improved  delivery  time  and  performance  of  military  in¬ 
tegrated  circuits  with  the  development  and  insertion  of  military-qualified  VHSIC 
chips  into  weapon  systems.  This  early  approach  to  a  military  hardware  problem 
employed  many  of  the  product  line  principals  explained  later  in  the  research  and 
applied  to  the  software  challenge  to  fully  use  the  hardware  capabilities, 

•  The  Software  Technology  for  Adaptable  Reliable  Systems  (STARS)  established  in 

1983  by  the  Department  investigated  methods  to  reduce  software  development 
costs,  increase  software  system  reliability,  improve  software  automation  techniques, 
and  identify  the  advantages  of  software  reuse, 

•  The  Department  contracted  with  the  SEI,  located  at  Camegie-Mellon  University,  in 

1984  to  investigate  new  software  technology,  analyze  software  development 
environments,  and  provide  software  and  system  engineering  education  [BBG+03], 

•  Finally,  several  Defense  Science  Boards  task  forces  and  summer  studies  since  1980 
reviewed  multiple  defense  programs  and  recommended  changes  to  the  Depart¬ 
ment’s  attitudes,  policies,  and  practices  regarding  software  development  and  acqui¬ 
sition  [CHK90]. 

Today,  many  of  the  Department’s  software-intensive  systems  lack  quality 
[HNC+00],  while  software  costs  identified  by  [Coh94,  SG94,  SG99,  RJI02]  continue  to 
increase.  The  persistent  systemic  issues  of  late  software  deliveries;  coupled  with  poor- 
quality  continues  to  grow  [Bow02,  Mic02a,  Mic02b];  and  stymie  the  demand  for  reliable 
software  systems.  [KMOla]  confirmed  a  causal  link  between  simulation  credibility  and  the 
quality  of  the  software  engineering  processes  and  practices  involved  in  simulation  software 
development  and  maintenance  practices  identified  in  many  of  the  earlier  studies  and  re¬ 
ports.  Achieving  simulation  credibility  and  confidence  in  M&S  results  also  depends  on  the 
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quality  and  interoperability  in  the  underlying  software,  hardware,  networks  and  software 
development  processes  and  products. 

A  preponderance  of  research  indicates  software  engineering  maturity  lacks  disci¬ 
pline  and  lags  significantly  behind  other  engineering  disciplines.  Our  research  suggests 
that  simulation  software  development  may  be  more  complex  than  the  development  of  other 
types  of  software,  since  a  model  is  an  “abstraction178  or  approximate  representation  of 
something’’  [GMS+96],  where  no  model  is  ever  completely  representative  [BCN96],  and 
[Sha75]  contends  no  model  is  absolutely  correct.  Disciplined  development  and  the  M&S 
verification  and  validation  process  is  critical  to  the  establishment  of  M&S  software  credi¬ 
bility  and  the  establishment  of  confidence  in  the  accreditation  decision  and  the  simulation 
results.  To  date,  the  Department  has  not  achieved  any  long-lasting  Department-wide  quan¬ 
tifiable  results  indicating  improved  simulation  credibility. 

Research  suggests  that  quality  simulation  software,  based  on  a  disciplined  software 
engineering  foundation  [Sha93,  VFS+99]  supports  simulation  credibility  and  the  mainte¬ 
nance  of  user  confidence  in  the  results.  However,  documented  systemic  simulation  soft¬ 
ware  engineering179  issues  [Arm64,  PAD78,  Che86a,  HW97,  GFOO,  CraOla,  RieOl]  since 
the  mid-1960s  included  the  lack  of  adequate  software  requirements,  software  configuration 
management,  verification  and  validation  processes,  software  engineering  processes,  and 
software  quality  methods.  Summarizing  the  simulation  software  portion  of  the  Depart¬ 
ment’s  software  Tower  of  Babel,  [BTE+93]  stated, 

Software  will  continue  to  be  a  problem  because  it  is  here  that  all  of  system 
complexity  hides.  The  software  problem  will  continue  to  be  the  problem  of 
exactly  what  does  the  system  do  [BTE+93]. 

Simulation  software  development  also  shares  all  the  same  cost,  performance, 
schedule,  and  quality  software  development  challenges  [Gib94,  Coh94,  SG94,  SG96c, 
SG99,  NJLOO,  BW01]  found  in  other  domains,  and  adds  the  complexity  of  developing  and 
validating  conceptual  models  with  credible  model  representations  of  reality  to  achieve  con- 


178  Abstraction  is  the  selective  emphasis  on  detail:  specific  details  are  suppressed  and  those  pertinent  to  the  problem  are  emphasized. 
Abstraction  mechanisms  focus  on  high-level  aspects  of  an  entity  while  concealing  details.  Three  commonly  used  abstraction  mecha¬ 
nisms  are  classification,  aggregation,  and  generalization  [Sow88]. 

179  ... 

Software  engineering  is  the  application  of  a  systematic,  disciplined,  quantifiable  approach  to  the  development,  operation  and  mainte¬ 
nance  of  software;  that  is  the  application  of  engineering  to  software  [BCC+Olb]. 
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fidence  in  M&S  results.  In  response  to  these  issues,  the  Department  initiated  three  large, 
joint  software-intensive  simulation  development  projects  during  the  1 990’s:  Joint  Modeling 
and  Simulation  System  (JMASS),  Joint  Simulation  System  (JSIMS),  and  Joint  Warfare 
System  (JWARS).  The  Department  approved,  funded  and  initiated  the  three  systems  to 
replace  Services  and  Agencies  legacy  M&S,  with  planned  objectives  of  meeting  software 
development  cost,  performance,  and  schedule  milestones,  improving  system  interoperabil¬ 
ity,  reducing  life-cycle  maintenance  costs,  and  conforming  to  the  new  HLA  standards. 

The  Joint  Modeling  and  Simulation  System  (JMASS)  [Bro92,  MeyOl]  started  in 
1990.  JMASS  is  unlike  other  Department  simulation  software  development  programs, 
which  plan  to  deliver  a  complete,  integrated  simulation.  The  JMASS  program  provides 
architectural  components  consisting  mostly  of  the  simulation  engine  and  services,  interface 
standards,  and  a  set  of  GUI-based  tools  to  build  models  to  integrate  the  models  with  other 
architectural  components  and  develop  a  simulation  [MeyOl].  The  major  deliverables  ex¬ 
pected  from  the  JMASS  effort  include  common  digital  simulation  architecture;  definition 
of  standard  interfaces;  application  of  commercial  standards;  a  CASE  tool  environment  and 
M&S  support  for  the  entire  acquisition  lifecycle. 

The  Joint  Simulation  System  (JSIMS)  [Mar97],  conceived  as  the  primary  Depart¬ 
ment  simulation  tool  supporting  future  joint- and  Service-based  training,  education,  and 
mission  rehearsal  roles  and  functions,  began  with  an  approved  July  1994  mission  need 
statement.  The  JSIMS  design  envisioned  a  single,  distributed,  integrated  simulation  envi¬ 
ronment  supported  by  a  core  infrastructure  and  mission  space  objects  maintained  in  a 
common  repository.  However,  by  June  1999,  [Ett99]  declared  that  JSIMS  had, 

“Serious  funding,  technical,  and  management  problems... JSIMS  remains  a 
troubled  program  ”,  with  major  architecture  issues,  and  an  inadequate 
management  structure,  requiring  a  major  new  baseline  of  the  acquisition 
program,  increased  Department  management  oversight  required,  and  the 
need  for  continued  Department  program  support”  [Ett99]. 

The  Initial  Operating  Capability  (IOC)  for  JSIMS,  originally  scheduled  for  Decem¬ 
ber  1999,  experienced  several  delays  with  the  last  JSIMS  IOC  scheduled  for  March  2003. 

180  The  JSIMS  PEO  declared  that  (1)  the  current  program  was  unexcutable;  (2)  the  acquisition  program  baseline  was  breached  and  must 
be  rebaselined;  and,  (3)  the  IOC  for  JSIMS  would  slip  at  least  1 1  months  [Ett99] 
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The  life-cycle  program  funding  profile  projects  $1.6  billion  for  the  period  of  1996  through 
2007  [UMS+Ola],  The  JSIMS  future  is  unclear  according  to  [UMS+Olb]  citing  a  lack  of 
assurance  that  the  Department  JSIMS  Milestone  Decision  Authority  would  be  able  to  make 
future  informed  investment  decisions  on  JSIMS,  based  on  reoccurring  historical  problems 
including  multiple  schedule  delays,  performance  issues,  and  increased  costs. 

The  Department  approved  a  third  major  simulation  development  effort,  the  Joint 
Warfare  System  (JWARS)  [MetOO,  ManOl],  in  May  1995  to  develop  a  state-of-the-art, 
multi-sided  simulation  of  joint,  campaign  level  warfare  representing  joint  functions,  proc¬ 
esses,  doctrine,  and  component  warfare  operations  of  future  warfare  to  support  analysis. 
The  JWARS  high-level  design  objectives  included  plans  to  represent  future  warfare  capa¬ 
bilities  supporting  force  structure  decisions,  course  of  action  analysis  for  the  Combatant 
Commanders,  and  the  development  of  operational  plans  from  a  joint  operation  perspective. 

The  JWARS  system  objectives  also  included  the  need  to  provide  a  better  assess¬ 
ment  of  C4ISR  contributions,  weapon  systems,  joint  requirements  and  the  analysis  of  mo¬ 
bility,  logistic  and  contingency  operations  impacts  on  warfighting  capability,  and  support 
for  the  analysis  of  investment  alternatives.  When  software  development  began  in  April 
1997,  the  JWARS  program  did  not  require  a  fonnal  conceptual  model  product,  and  subse¬ 
quently  the  V&V  agent  created  a  virtual  conceptual  model  from  available  development  ar¬ 
tifacts  [MetOO].  Initial  V&V  efforts  to  include  algorithm  validation  were  unsuccessful  ac¬ 
cording  to  [MetOO].  The  lack  of  a  simulation  conceptual  model  has  been  a  recurring  sys¬ 
temic  issue  with  Department  simulation  software  development,  a  major  reason  for  the  cur¬ 
rent  lack  of  simulation  and  federation  credibility  and  general  skepticism  of  the  Depart¬ 
ment’s  simulation  results. 

The  three  major  Department  simulation  software  engineering  efforts:  JMASS, 
JSIMS,  and  JWARS  are  all  over  cost,  behind  schedule,  and  have  yet  to  demonstrate  they 
can  achieve  performance  objectives.  Partly  based  on  these  experiences,  the  Department 
implemented  a  new  approach  for  the  Joint  Distributed  Engineering  Plant  (JDEP)  initiative 
to  improve  System-of-System  interoperability:  large-scale  reuse  [DahOla,  DahOlb,  DTOla, 
DTOlb,  Rad02].  [WolOl]  established  the  Department  policy  for  mission  critical  C2  inter¬ 
operability  for  the  Joint  Task  Force  and  below,  including  the  JDEP.  Rather  than  build  the 
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JDEP  from  scratch,  the  Department  is  making  use  of  large-grain  reuse  [CWS02],  pulling 
the  necessary  assets  from  across  the  Department’s  enterprise  to  support  integration,  inter¬ 
operability,  and  meet  information  assurance  objectives. 

Software  engineering  and  the  V&V  processes  support  improved  credibility  and  con¬ 
fidence.  [KMOla]  identified  the  link  between  simulation  credibility  and  software  engineer¬ 
ing  processes  and  good  model  management  practices.  The  Department’s  verification  proc¬ 
ess  also  “evaluates  the  extent  to  which  the  model  or  simulation  has  been  developed  using 
sound  and  established  software  engineering  techniques”  [DoD98,  DoD95,  AR97].  Stan¬ 
dard  software  development  processes  and  software  engineering  methodologies  have  been 
used  to  develop  Department  simulation  software,  including  the  incremental,  prototype,  evo¬ 
lutionary,  spiral  [Boe93,  BoeOO],  and  re-engineering  methods  [RPGOO].  However,  De¬ 
partment  software  development  processes  for  the  most  part,  lack  the  discipline  of  the  more 
mature  physical  engineering  fields  [DoDOla],  and  also  lack  consistent  enforcement  mecha¬ 
nisms  [HNC+00].  This  statement  may  be  significant  since  troubled  or  failed  software  pro¬ 
jects,  industry-wide,  are  still  the  norm,  and  the  costs  for  failure  are  staggering. 

Software,  as  a  product  and  the  technology  for  delivering  a  product,  has  made  major 
contributions  to  the  twenty-first  century  world,  but  major  systemic  challenges  continue  to 
persist  according  to  [Rei93a,  Rei93b],  and  many  of  these  problems  may  cause  unintended 
consequences.  While  there  has  been  marked  progress  in  many  areas, 

Many  prior  deficiencies  continue  to  shape  DoD  M&S  because  they  require 
long-term  solutions  and  competing  priorities  that  have  slowed  progress 
[DoDOla], 

[Kil02]  cites  the  lack  of  cooperation  from  M&S  developers  as  the  “single  greatest 
obstacle  to  developing  a  meaningful  assessment  of  the  credibility  and  fitness  of  simula¬ 
tions”  [Kil02].  As  a  result,  [PacOlb]  considers  the  state  of  current  practices  for  simulation 
software  development,  associated  M&S  processes,  and  product  management  inadequate. 
Although  [Pre97]  suggests  we  have  a  chronic  affliction  as  opposed  to  a  software  crisis  , 


Fitness:  Providing  the  capabilities  needed  or  being  suitable  for  some  purpose,  function,  situation  or  application  [RPGOO]. 

The  term  “software  crisis”  was  identified  by  the  DoD  in  the  mid-1970s  [SCC+98]  and  has  been  associated  with  the  complex  prob¬ 
lems  associated  with  producing  software  that  was  developed  on  time,  within  budget,  operated  properly,  and  was  maintainable  over  its  life 
cycle.  More  recently  its  was  identified  by  [Gib94]  as  the  need  for  a  mature  software  engineering  discipline  supporting  an  information- 
age  society,  that  he  believes  remains  years,  maybe  decades  away  from  achieving,  even  after  fifty  years  of  evolution  [Gib94]. 
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he  cites  an  aging  software  plant  and  several  unresolved  systemic  software-related  problems 
including: 

•  Hardware  advances  continue  to  outpace  software  capabilities, 

•  Software  has  not  kept  up  with  the  demand  for  new  or  replacement  programs, 

•  Society  is  now  dependent  on  the  reliable  operation  of  software  as  witnessed  by  the 
Year  2000  problem, 

•  High  software  reliability  and  quality  have  not  been  achieved, 

•  Poor  design  and  inadequate  resources  [Pre97]. 

The  software  industry  is  also  growing  ever  more  important  to  the  twenty-first  cen¬ 
tury  society.  The  demand  for  new  software  has  grown  so  rapidly  that  the  demand  far  ex¬ 
ceeds  the  supply  and  the  following  conditions  prevail  according  to  [JBC+99]: 

•  Software  is  fragile,  unreliable,  difficult  and  labor-intensive  to  develop,  test  and 
evolve, 

•  The  Nation’s  ability  to  produce  software  does  not  meet  the  demand, 

•  The  Nation  depends  on  fragile,  inadequate,  software  products  developed  with  im¬ 
mature  processes, 

•  Current  technologies  to  support  reliable  and  secure  software  are  inadequate, 

•  Software  continues  to  grow  more  sophisticated  and  diverse, 

•  Software  is  replacing  manual  processes  in  many  activities  previously  exercised 
manually  by  individuals, 

•  The  Nation  needs  to  invest  more  in  software  research  [JBC+99]. 

Historically,  the  software  engineering  community  viewed  software  engineering 
processes  as  discrete  activities.  However,  while  there  are  discrete  activities  including  en¬ 
trance  and  exit  criteria,  design,  coding,  and  testing,  most  of  the  development  process  pro¬ 
gresses  in  a  continuous  fashion  [STS99].  The  continuous  software  engineering  activities 
[STS99]  include  configuration  management,  quality  assurance,  verification  and  validation, 
maintenance  planning,  risk  analysis  and  management,  and  software  engineering  project 
management. 

Software  requirements  for  models,  simulations,  and  federations  include  three  major 
categories:  the  problem  domain,  the  user  domain,  and  the  situation  domain  [RPG00].  The 
simulation  software  requirements  process  articulated  by  [BL91,  Som95,  SG96a,  SG96b, 
BIL97,  Pre97,  BW97,  CS99,  HHP00,  KG00,  MWC00,  RPG00,  HL01]  at  the  most  basic 
level  are  properties,  exhibited  by  a  new  or  adapted  M&S  system  to  solve  a  particular  prob- 
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lem.  They  may  have  different  properties  such  as  process,  or  system  constraint,  and  product 
parameters,  which  include  functional  (capability)  and  non-functional  requirements  such  as 
constraints  and  quality  requirements.  For  instance,  [Som95]  suggests  a  reliability  require¬ 
ment  places  constraints  on  the  system  architecture. 

Emerging  requirements  may  be  dependent  upon  the  system  interoperation,  as  op¬ 
posed  to  a  single  feature  or  component,  and  critically  dependent  on  system  architecture  de¬ 
cisions.  Requirements  elicitation  is  seldom  easy  and  requires  familiarity  with  many  tech¬ 
niques  and  an  understanding  of  the  impact  of  social,  political,  or  economic  factors  on  the 
different  stakeholders’  view  of  the  requirements.  Many  times  requirements  understanding 
continue  to  evolve  during  the  design  and  development  phases.  The  requirements  then  un¬ 
dergo  analysis  to  understand  the  system  components  and  how  they  interact  with  other  com¬ 
ponents,  establish  baselines  and  prioritize  the  requirements,  and  finally  develop  the  esti¬ 
mated  cost.  The  root  cause  in  the  Ariane  5  disastrous  failure  was  related  to  a  breakdown  in 
the  requirements  process  [STS99]. 

Furthermore,  software  engineering  processes,  development  and  maintenance  prac¬ 
tices  and  procedures  may  provide  significant  insights  into  M&S  credibility  [KMOla],  It  is 
also  important  to  review  M&S  in  the  context  of  the  software  engineering  environment.  As 
the  Department  evolves  from  procedural-based  software  development  techniques  to  an  ob- 
ject-oriented  approach,  [Mey98]  suggests  that  a  model  may  be  defined  as  a  “digital  or 
software  representation  of  a  physical  entity”,  in  an  object-oriented  paradigm  comprised  of 
entities  with  associated  attributes  and  behaviors,  while  a  simulation  may  be  viewed  as  the 
“software  framework  or  architecture  within  which  physical  entities  are  animated”  [Mey98], 
At  Figure  7-1  is  one  version  the  evolving  software  engineering  body  of  knowledge  (SWE- 
BOK)  [Moo99,  BCC+Olb]  supporting  improved  simulation  credibility. 


Object-oriented  is  a  method  of  implementation  in  which  programs  are  organized  as  cooperative  collections  of  objects,  each  of  which 
represents  an  instance  of  some  class,  and  whose  classes  are  all  members  of  a  hierarchy  of  classes  united  by  inheritance  relationships 
[PMR94].  Includes  object-oriented  analysis,  design  and  programming. 
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Figure  7- 1 .  The  S WEBOK  (From  [BCC+0 1  b]) 

The  software  engineering  process  knowledge  area  is  a  dynamic,  rapidly  evolving 
field,  with  new  paradigms  and  models  constantly  introduced.  The  Department’s  approach 
to  software  evolved  from  a  software  developer  to  a  software  acquirer,  and  today  the  De¬ 
partment  acquires  most  software  as  commercial-off-the-shelf  products  or  contracts  with  the 
private  sector  for  the  development  effort.  However,  [Mos02]  cautions  that  government 
program  managers  developing  the  contract  request  for  proposal  and  the  government  con¬ 
tractor  world  winning  the  contract  awards  do  not  understood  commercial  best  practices. 

Conversely,  [Mos02]  contends  the  real  commercial  world  contractors,  bidding  for 
private  sector  contracts  know  that  best  practices  critical  for  success  includes  quality  soft¬ 
ware,  software  architectures,  and  product  lines,  which  provide  a  competitive  advantage. 
[Mos02]  concern  echoes  a  recent  Defense  Science  Board  [HNC+00]  report,  which  stated, 
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The  major  factor  in  successful  software  development  is  disciplined  execu¬ 
tion... too  often;  programs  lacked  well  thought-out  disciplined  program 
management  and  /  or  software  development  processes.  [HNC+00]. 

At  a  macro  level,  [FerOl]  notes  that  software  product  quality  and  progress  in  the 
Department’s  weapon  systems  depend  on  a  myriad  of  complex  internal  and  external  forces 
(e.g.,  Congress,  DoD,  Services),  both  within  and  outside  the  program  manager’s  purview 
[Bro74].  In  an  article  by  [JonOO]  on  the  best  and  worst  software  development  practices,  he 
characterized  the  military  software  domain  with  good  development  methods,  supported  by 
good  quality  control,  but  executing  “marginal  or  deficient  project  management  method” 
[JonOO],  However,  there  are  some  positive  indicators.  [SG99]  cites  improved  project  man¬ 
agement,  processes,  and  people  as  contributors  to  this  positive  trend,  while  interestingly 
enough,  technology  was  not  the  problem,  n  or  the  silver  bullet  [Bro87,  Bar02c]  solution. 
[SG99]  noted  three  factors  contributing  to  this  positive  trend:  smaller  projects,  better  pro¬ 
ject  management  and  greater  use  of  standard  infrastructures. 

Future  technical,  organizational,  cultural,  and  managerial  factors  in  the  missile  do¬ 
main  simulation  development  domain  cited  in  Table  7-1  will  be  significant  challenges.  In¬ 
stitutional  change  or  transformation,  led  by  early  adapters  requires  reengineering  of  many 
institutions  and  processes,  especially  in  the  Department  of  Defense  with  its  current  and  fu¬ 
ture  strategy  to  defend  the  Nation  built  around  information.  Where  once  the  prevailing 
school  of  thought  in  the  Department’s  infonnation  technology  world  centered  on  process¬ 
ing  power  and  bandwidth  issues,  these  problems  have  technical  solutions  and  are  now  more 
a  matter  of  implementation.  Replacing  these  older  problem  sets  are  many  new,  complex 
issues  including  information  operations,  information  assurance,  and  the  security  of  personal 
infonnation  from  misuse. 

Table  7-1  summarizes  the  Department  institutional  constraints  adversely  affecting 
M&S  credibility  today.  Note  the  close  conelation  of  the  constraints  cited  in  Table  7-1  to 
the  major  research  areas  identified  in  Figure  2-2.  Significant  persistent,  organization,  proc¬ 
ess,  management,  and  technical  constraints  limit  or  adversely  affect  simulation  software 
credibility  in  the  Department  [DoD95,  San97b,  Ett99,  RPGOO,  HNC+00,  PacOlb,  DoD03a, 
Nor03].  Software  engineering  and  software  architecture  skills  are  now  emerging  as  critical 
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domain  disciplines  that  must  continue  to  mature  the  corresponding  bodies  of  knowledge,  if 
the  Department  and  the  Nation  are  to  exploit  the  full  potential  of  computer  technology,  and 
set  the  foundation  for  the  next  great  era  in  computing,  the  Product  Line  Era.  Product  Line 
Practice  [Nor03]  areas  address  many  of  these  constraints. 


Department  Model  &  Simulation  Credibility  Constraints  | 

SOURCE 

CONSTRAINT  I 

CONSTRAINT  II 

CONSTRAINT  III 

M&S  |DoD95,  RPGOO] 

TECHNICAL 

CULTURAL 

MANAGEMENT 

SBA  |San97b] 

PROCESS 

CULTURAL 

ENVIRONMENT 

JSIMS  [Ett99] 

TECHNICAL 

ARCHITECTURE 

MANAGEMENT 

DEF  SCI  BOARD  [HNC+00] 

PROCESS 

DISCIPLINE 

MANAGEMENT 

VALIDATION  [PacOlb] 

PROCESS 

DEVELOP  SOFTWARE 

MANAGEMENT 

M&S  STRATEGY  [DoD03a] 

PROCESS 

ORGANIZATIONAL 

BUSINESS 

PRODUCT  LINE  |Nor03] 

SOFTWARE  ENGR 

ORGANIZATIONAL 

MANAGEMENT 

Table  7-1.  Department  Institutional  Constraints  Affecting  M&S  Credibility 

Future  domain  software  engineers  and  domain  software  architects  will  need  to  un¬ 
derstand  the  viewpoints  of  multiple,  diverse  stakeholders  at  all  levels  of  the  Department 
Enterprise  and  synchronize  the  software  architecture  and  software  engineering  solution  that 
meets  cost,  performance,  schedule,  and  other  constraints  cited  in  Table  7-1.  Developing 
and  maintaining  currency  in  the  skill  sets  of  domain  software  engineers  and  software  archi¬ 
tects  grounded  in  the  technical  foundations  of  the  field  (e.g.,  computer  science,  software 
engineering,  software  architecture,  infonnation  technology),  with  experience  in  multiple- 
domains  and  functional  areas  will  also  require  continuous  education  opportunities  and  pro¬ 
gressive  developmental  assignments. 

E.  SOFTWARE  ENGINEERING  EDUCATION  INITIATIVES 

The  Department’s  draft  strategic  M&S  plan  [DoD03a]  includes  the  a  strategic  goal 
for  awareness,  education,  training,  and  collaboration,  and  proposes  M&S  education  pro¬ 
grams  through  the  postgraduate  level,  providing  Department  employees  with  hands-on  ex¬ 
perience  through  on-the-job-training,  internships,  and  rotational  assignments.  Systemic 
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information  technology  (IT)  issues  facing  the  Department  today  include  the  shortage  of 
educated  IT  professionals  [HP99a,  PH99,  BynOO,  MatOO,  RBB+00],  with  a  special  empha¬ 
sis  in  the  area  of  information  assurance  [Ste02b]. 

Improving  the  Department’s  IT  competencies  includes  many  disciplines  and  emerg¬ 
ing  bodies  of  knowledge,  including  software  engineering.  A  growing  body  of  literature 
addresses  the  complexity  of  the  software  engineering  field  including: 

•  Studies  of  the  best  software  engineering  training  practices  [MTC96,  Wie96b], 

•  Software  engineering  education  philosophy  and  alternative  approaches  [Par99, 
Bec99,  BerOOb,  CDC+00,  HisOO,  RBB+00,  ShaOOb,  RieOl,  CT02,  BM02,  HH02a, 
HH02b,  SBM02,  DT02,  Jen02,  BK02b], 

•  Teaching  software  engineering  processes  [Wie96b,  BKOlb,  BCH+02,  BKK02], 

•  Collaborative  and  distributed  software  engineering  education  to  transform  surplus 
engineers  from  traditional  areas  into  software  engineers  [BPD02,  HH02b,  HMR02, 
UHC02], 

•  Applying  the  evolving  software  body  of  knowledge  [Wie96b,  TH02],  and 

•  Adding  a  software  engineering  capability  to  existing  military  system  engineering 
disciplines  [Flo02b]. 

Emerging  from  the  intellectual  stimulus  and  software  engineering  studies  are  direc¬ 
tories  of  industry  and  university  software  engineering  collaboration  programs  [Bec99], 
guidelines  for  software  engineering  education  [BHH+99],  certification  programs  [OSA+Ol, 
computing  curricula  guidelines  including  software  engineering  [CDC+00],  and  new  gradu¬ 
ate-  and  postgraduate-level  software  engineering  education  opportunities  [NPS02]. 

F.  SUMMARY 

In  Chapter  VII  we  reviewed  three  selected  areas  influencing  credibility  of  large- 
scale,  legacy  simulations  in  the  Department  including  process  improvement,  the  status  of 
new  major  simulation  software  engineering  initiatives  (e.g.,  JWARS,  JSIMS,  JMASS)  and 
the  growing  software  engineering  body  of  knowledge.  The  chapter  also  included  a  discus¬ 
sion  of  the  challenges  generated  by  Department  contract  management  oversight  for  achiev¬ 
ing  quality  software  and  reducing  heterogeneous  system  representation  anomalies,  Depart¬ 
ment  institutional  factors  affecting  M&S  credibility,  and  recent  Department  software  engi¬ 
neering  education  initiatives.  In  addition,  research  reveals  a  growing  awareness  that  the 

Department  needs  qualified  software  engineers  who  understand  the  foundations  of  the 
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software-intensive  systems  upon  which  we  basing  our  future  security,  economic  prosperity, 
and  military  preparedness,  including  credible  M&S. 

The  first  component  supporting  M&S  credibility  is  process  improvement.  By  1996, 
American  corporate  information  technology  project  management  realized  no  technological 
or  silver  bullet  solution  existed  to  reverse  the  tide  of  software  project  failures.  The  true 
scope  of  the  problem  and  solution  included  processes  and  people.  In  an  effort  to  improve 
quality,  process  improvement  models  including  the  Software  Engineering  Institute  Capabil¬ 
ity  Maturity  Model  Integration  (CMMI),  Capability  Maturity  Model  (CMM)  for  software, 
Software  Acquisition  CMM,  Systems  Engineering  CMM,  ISO  9000,  and  the  draft  ISO 
15504  standard  for  software  process  assessment  (SPICE)  initiatives  evolved.  These  proc¬ 
ess  models  matured  based  on  the  premise  that  the  quality  of  the  processes  used  to  develop 
and  maintain  the  software  significantly  influenced  the  quality  of  a  software  system.  Initial 
indications  suggest  that  process  improvement  methods  contribute  to  improve  overall  soft¬ 
ware  product  quality  and  productivity.  However,  the  Department  inconsistently  supports 
software  process  improvement. 

Contract  management  oversight  is  a  critical  process  for  an  organization  acquiring 
software.  The  challenges  of  managing  the  myriad  of  contracts  supporting  the  Department’s 
IT  initiatives,  including  M&S  software  support,  and  improve  collaboration,  continue  to 
grow.  Federal  Government  spending  on  IT  services  almost  doubled  from  fiscal  years  1997 
to  2001,  increasing  from  $9  billion  to  $17  billion,  with  the  Department  of  Defense  remain¬ 
ing  the  largest  single  acquirer  of  IT  services  and  increasing  spending  by  about  forty-one 
percent  throughout  the  period.  During  the  same  period  spending  on  IT  services  from  the 
General  Service  Administration  (GSA)  greatly  increased  with  spending  on  the  GSA  federal 
supply  service  schedule  growing  from  approximately  $405  million  to  $4.3  billion. 

Research  indicates  software  engineering  maturity  lacks  discipline  and  lags  signifi¬ 
cantly  behind  other  engineering  disciplines.  Our  research  also  suggests  that  simulation 
software  development  may  be  more  complex  than  the  development  of  other  types  of  soft¬ 
ware,  since  a  model  is  an  “abstraction  or  approximate  representation  of  something” 
[GMS+96],  where  no  model  is  ever  completely  representative  [BCN96],  and  [Sha75]  con¬ 
tends  no  model  is  absolutely  correct.  Disciplined  development  and  the  M&S  verification 
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and  validation  process  is  critical  to  the  establishment  of  M&S  software  credibility  and  the 
establishment  of  confidence  in  the  accreditation  decision  and  the  M&S  results.  However, 
the  Department  has  not  achieved  any  long-lasting  Department-wide  results  improving 
simulation  credibility.  Quality  simulation  software,  based  on  a  disciplined  software  engi¬ 
neering  foundation  [Sha93,  VFS+99]  supports  simulation  credibility  and  the  maintenance 
of  user  confidence  in  the  results.  However,  documented  systemic  simulation  software  en¬ 
gineering  issues  [Arm64,  PAD78,  Che86a,  HW97,  GFOO,  CraOla,  RieOl]  since  the  mid- 
1960s  included  the  lack  of  adequate  software  requirements,  software,  configuration  man¬ 
agement,  verification  and  validation,  software  engineering,  processes,  and  software  quality. 

The  next  component  of  this  research  supporting  M&S  credibility  is  a  review  of  the 
case  studies  of  three  modern  software  engineering  projects  involving  new  Department 
M&S  initiatives:  JWARS,  JSIMS,  and  JMASS.  This  review  was  significant  for  several 
reasons.  First,  these  replacement  systems  are  overdue.  The  new  Joint  M&S  programs 
JWARS,  JSIMS,  and  JMASS,  scheduled  to  replace  many  legacy  systems  are  also  over  cost, 
and  have  reduced  the  original  number  of  requirements  and  requirements  to  meet  a  future 
operational  status.  It  is  still  to  be  detennined  if  they  will  be  useful  for  the  emerging  and 
rapidly  evolving  requirements  of  the  21st  century. 

Improving  the  Department’s  IT  competencies  includes  many  disciplines  and  emerg¬ 
ing  bodies  of  knowledge,  including  software  engineering.  A  growing  body  of  literature 
addresses  the  complexity  of  the  software  engineering  field.  The  Department’s  draft  strate¬ 
gic  M&S  plan  [DoD03a]  includes  the  a  strategic  goal  for  awareness,  education,  training, 
and  collaboration,  and  proposes  M&S  education  programs  through  the  postgraduate  level, 
providing  Department  employees  with  hands-on  experience  through  on-the-job-training, 
internships,  and  rotational  assignments.  However,  systemic  information  technology  (IT) 
issues  facing  the  Department  today  include  the  shortage  of  educated  IT  professionals 
[HP99a,  PH99,  BynOO,  MatOO,  RBB+00],  with  a  special  emphasis  in  the  area  of  informa¬ 
tion  assurance  [Ste02b]. 

For  over  twenty  years  the  Department  hoped  for  a  “silver  bullet”  solution  to  resolve 
the  “software  crisis”.  However,  as  hope  for  a  “silver  bullet”  technical  breakthrough  to  im¬ 
prove  the  Department’s  software,  including  simulation  software,  waned  in  the  1990s,  it  be- 
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came  apparent  that  other  non-technical  solutions  were  required.  A  second  underlying  cause 
for  the  current  situation  is  the  apparent  dichotomy  presented  by  the  similar  M&S  factors 
cited  in  Table  7-1  concurrently  identified  as  the  problem  and  the  solution.  In  1996  a  major 
Department  M&S  study  [PHP+96]  identified  three  major  types  of  systemic  Department 
M&S  challenges,  technical,  cultural,  and  managerial;  while  concurrently  the  Simulation- 
Based  Acquisition  (SBA)  initiative  identified  three  similar  factors,  process,  culture,  and 
environment  as  SBA  enablers.  More  recently  the  SEI  endorsed  three  related  practice  areas, 
technical  management,  organizational,  and  software  engineering  to  support  the  successful 
implementation  of  software  product  lines. 

These  factors  are  significant  since  this  methodology  represents  an  institutional  re¬ 
alization  that  the  resolution  to  the  twenty  year  old  “software  crisis”  is  not  a  “silver  bullet” 
solution.  Instead,  the  problem  and  the  solution  are  multi-dimensioned,  complex  issues  that 
require  changes  in  all  sectors  of  the  Department.  This  change  or  transfonnation,  led  by 
early  adapters  requires  a  reengineering  of  many  institutions  and  processes,  especially  in  the 
Department  with  its  current  and  future  strategy  to  defend  the  Nation  built  around  informa¬ 
tion.  Where  once  the  prevailing  school  of  thought  in  the  information  technology  world 
centered  on  processing  power  and  bandwidth  issues,  these  problems  are  technically  solved 
and  more  a  matter  of  implementation. 

This  realization  will  take  time  for  society  and  the  Department  to  truly  assimilate  and 
adapt  to  during  the  ongoing  transition  phase  into  the  Infonnation  Age.  We  must  however, 
also  be  mindful  of  the  many  Industrial  Age  vignettes  where  nations,  armies,  navies,  or  air 
forces  were  slow  to  adapt  and  the  severe  consequences  of  that  that  failure  in  terms  of  defeat 
or  loss  of  life. 


180 


VIII.  SOFTWARE  ENGINEERING  OPTIONS  TO  IMPROVE  CREDIBILITY 


A.  INTRODUCTION 

Chapter  VIII  discusses  traditional  product  lines,  and  introduces  software  product 
lines  and  software  architectures.  The  chapter  also  reviews  component  development  sup¬ 
porting  a  product  line  methodology,  product  line  practice  areas,  and  an  overview  of  evolv¬ 
ing  software  architecture  theory  potentially  applicable  to  improved  M&S  credibility  and 
reducing  heterogeneous  system  representation  anomalies. 

B.  PRODUCT  LINES 

Historically,  Industrial  Age  companies  found  that  using  common  assets  to  build  re¬ 
lated  systems  yielded  significant  market  opportunities  and  improvements  in  time  to  market, 
customer  satisfaction,  product  quality,  meaningful  metrics,  and  supported  a  competitive 
advantage  for  additional  market-share.  Henry  Ford  adopted  the  same  concept  when  he  in¬ 
troduced  the  assembly  line,  or  product  line  method  to  the  automobile  industry,  establishing 
the  private  industry  product  line  model  [MNJ+02], 

In  possibly  the  largest  historical  application  of  the  product  line  concept,  the  United 
States  private  industry  in  World  War  II  manufactured  the  majority  of  the  ships,  planes, 
tanks,  and  other  major  end  items  for  the  Allied  war  effort  in  numbers  that  the  Axis  powers 
could  never  match,  establishing  the  same  competitive  advantage  for  wartime  victory. 
Product  lines  continued  to  evolve  in  many  Fortune  500  companies  including  Ford,  McDon¬ 
ald’s  and  Boeing,  albeit  differently.  In  the  Boeing  product  line  example,  the  parts  list  of 
two  entirely  different  aircraft,  the  757  and  767  models,  overlapped  by  approximately  sixty 
percent  [Cle99]. 

C.  SOFTWARE  PRODUCT  LINES 

As  the  Information  Age  evolved  and  the  corpus  of  software  core  assets  accumu¬ 
lated,  [Par76]  noted, 

We  consider  a  set  of  programs  to  constitute  a  family  whenever  it  is  worth¬ 
while  to  from  the  set  by  first  studying  the  common  properties  of  the  set  and 
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then  detennining  the  common  properties  of  the  individual  family  members 
[Par76]. 

The  software  product  line  concept  builds  on  the  experiences  of  component  reuse  including 

subroutines  in  the  1960s,  modules  in  the  1970s,  objects  in  the  1980s  and  component-based 

systems  in  the  1990’s.  The  SEI  developed  a  product  line  definition  from  the  successful 

private  industry  model  and  early  analysis  by  [Par76].  A  product  line  involves: 

A  set  of  products  sharing  a  common,  managed  set  of  features  that  satisfy 
specific  needs  of  a  selected  market  segment  or  mission  [BFG+00,  Nor03]. 

Software  product  lines  building  on  product  commonality  however  are  a  relatively 
new  concept.  The  SEI  Product  Line  Practices  Initiative  built  on  the  concept  of  a  product 
line  and  introduced  the  concept  of  software  product  lines  as  “a  set  of  software-intensive 
systems  sharing  a  common,  managed  set  of  features  that  satisfy  the  specific  needs  of  a  par¬ 
ticular  market  segment  or  mission  and  that  are  developed  from  a  common  set  of  core  as- 
sets  in  a  prescribed  way”  [CN02].  The  establishment  of  the  software  product  line  field¬ 
ing  approach  requires  core  asset185  development,  or  domain  engineering186,  and  product 
development,  also  called  application  engineering  using  these  core  assets  [BC96,  Jon99, 
Nor03]. 

As  an  organization’s  potential  strategic  infonnation  resource,  the  evolution  of  a 
software  product  line  requires  an  understanding  of  the  organization’s  strategic  plans,  goals, 
business  objectives,  culture,  technical  acumen,  risk  threshold,  and  life  cycle  management 
plans  [ISOOO].  This  analysis  supports  the  multiple  related  software  product  line  families 
and  products,  each  with  concurrent  versions  and  releases.  A  product  family  is  also  a  set 
of  products  built  from  a  common  set  of  core  assets  [Nor03,  CN02].  Although  a  product 
line  does  not  require  a  product  family,  and  a  product  family  does  not  necessarily  constitute 
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Core  assets  are  those  assets  that  form  the  basis  for  the  software  product  line.  Core  assets  often  include  but  are  not  limited  to,  the 
architecture,  reusable  software  components,  domain  models,  requirements  statements,  documentation  and  specifications,  performance 
models,  schedules,  budgets,  test  plans,  test  cases,  work  plans,  and  process  descriptions.  The  architecture  is  key  among  the  collection  of 
core  assets  [Nor03]. 

1 85  A  core  asset  is  a  software  artifact  used  in  the  production  of  more  than  one  product  in  a  software  product  line.  A  core  asset  may  be 
architecture,  a  software  component,  a  process  model,  a  plan,  a  document,  or  any  other  result  of  building  a  system  [Nor03]. 

The  domain  engineering  process  develops  software  assets  for  one  or  more  domains  [Nor03]. 

187  The  application  engineering  process  develops  software  products  from  partial  solutions  or  knowledge  embodied  in  software  assets 
[Nor03]. 

A  product  family  method  may  yield  the  greatest  efficiencies  for  developing  product  lines,  depending  upon  market  targets  or  feature 
relationships,  although  it  is  not  a  requirement  [Nor03]. 


182 


a  product  line,  the  SEI  definition  of  a  software  product  line  according  to  [Nor03,  CN02] 
implies  a  product  line  of  software  products  developed  as  a  product  family  in  a  prescribed 
way,  since  this  method  leverages  and  amortizes  prior  investment  [BC96,  BBW+00],  and 
potentially  produces  greater  efficiencies,  based  on  economies  of  scope  and  economies  of 
scale189. 

Software  product  lines  experience  and  growth  in  the  commercial  software  sector 
and  the  Department  provides  a  growing  body  of  software  product  line  case  studies  in  the 
following  areas:  product  line  investment  analysis  [Wit96,  BBW+00],  product  line  experi¬ 
ence  and  practice  [BC96,  WL99,  DonOO,  CN02],  product  line  organization  and  manage¬ 
ment  [BCC+97,  BCC+98,  BCN+98,  Cle99,  BCD+00,  CGF+00,  WapOO,  VWOO,  TCOOO, 
MNJ+02],  product  line  methods  [CJBOO,  KLD02,  KNOO],  product  line  process  [WooOOa, 
FHROO],  product  line  components  (reuse)  [Cle97,  JNR98,  ABMOO,  BCSOO,  PPOO,  GriOO, 
CN02],  product  line  architectures  [DSOO,  ProOO,  ShaOOa],  product  line  tools  and  techniques 
[CJBOO,  AOV+OO,  YMK+00,  SSP+00,  STOO,  HOF+OO],  product  line  domain  engineering 
[ADD+00,  LKK+00,  DagOO,  HSVOO,  TPOO,  MD02],  and  product  lines  in  the  Department 
[Jon99,  BFJ99,  BFG+00,  BOS02,  CDS02,  Cam02], 

[BFG+00,  DDW01,  CN02,  KLD02]  suggest  employing  a  software  product  line  at 
one  or  more  of  the  following  levels:  system190,  subsystem,  or  component,  depending  on  the 
application  domain,  degree  of  commonality,  and  feature  variability.  A  software  product 
line  notes  [Nor03]  includes  multiple,  related  products  with  product-specific  cycles  of  re¬ 
leases  and  versions,  which  evolve  in  consonance  with  the  product  line  as  a  whole.  The  up¬ 
front  cost  of  developing  the  first  software  product  line  member(s)  often  requires  a  signifi¬ 
cant  investment,  supported  by  a  business  case  and  market  analysis.  Fielding  a  software 
product  line  includes  the  development  of  core  assets  and  products  coordinated  by  manage¬ 
ment  activities.  Software  product  line  development  includes  mining  existing  products  for 
generic  assets  or  core  assets  for  later  development  use  in  an  iterative  manner,  based  on  cur¬ 
rent  needed  capabilities,  anticipated  future  requirements,  and  likely  future  product  variants 
[BOSOO,  CDK+01,  CN02]. 

189 

The  condition  where  fewer  inputs  such  as  effort  and  time  are  needed  to  produce  greater  quantities  of  a  single  output  (e.g.,  economy  of 

scale),  or  a  greater  variety  of  outputs  (e.g.,  economy  of  scope)  [Nor03]. 

190 

A  product  line  system  is  a  member  of  a  software  product  line  [Nor03]. 
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1. 


Potential  Benefit  of  a  Software  Product  Line 


A  product  line  approach  to  developing  and  deploying  software-intensive  systems 
offers  great  promise  according  to  [WL99,  BFG+00,  CN02]  for  delivering  higher  quality 
systems  in  a  shorter  time  and  at  reduced  cost.  According  to  analysis  of  the  industrial  sector 
experience  by  [BFG+00,  LKK+00,  DagOO,  HSVOO,  and  CN02]  a  product  line  approach  to 
software-intensive  systems  can  save  money  and  result  in  a  faster  time  to  field  quality  sys¬ 
tems.  [Jon99,  CN02]  cite  a  number  of  organizations  gaining  order-of-magnitude  improve¬ 
ments  in  efficiency,  productivity,  and  quality  through  a  product  line  approach.  [BC96, 
Jon99,  WL99,  BBW+00,  CN02]  also  note  that  even  more  important  than  cost  savings  is  the 
fact  that  product  line  practice  enables  an  organization  to  get  its  product  to  the  market  more 
rapidly.  As  an  example,  [BC96]  provides  a  case  study  of  a  successful  product  line  imple¬ 
mentation  by  the  CelsiusTech  Systems  AB  of  Sweden  in  the  area  of  large,  embedded,  real¬ 
time  shipboard  command  and  control  systems. 

The  SEI  identified  organizations  benefiting  from  software  product  line  practice  in 
workshops  cited  by  [BCC+97,  Cle97,  BCC+98,  BCN+98,  CW98,  BCD+00,  CGF+00]  and 
case  studies  [BC96]  including: 

•  The  Swedish  naval  defense  contractor,  CelsiusTech  experienced  a  favorable  rever¬ 
sal  in  the  hardware -to-software  cost  ratio  from  35:65  to  60:20,  now  favoring  soft¬ 
ware, 

•  Hewlett-Packard  has  metrics  reflecting  two  to  seven  times  cycle  time  improve¬ 
ments, 

•  Motorola  experienced  a  four  times  cycle  improvement  with  80%  reuse  on  a  pager 
product  line, 

•  Cummings  Engine  recorded  a  dramatic  reduction  in  one  case  for  a  system  build  and 
integration  from  about  one  year  to  three  days, 

•  Thompson-CSF  with  air  traffic  control  systems, 

•  Alltel  supporting  commercial  bank  systems, 

•  Ericsson,  Noki,  Lucent,  and  AT&T  in  communication  systems, 

•  Boeing  in  air  flight  software,  and 

•  The  National  Reconnaissance  Office  command  and  control  systems  for  satellites 
[BC96,  BCC+97,  Cle97,  BCC+98,  BCN+98,  WL99,  BCD+00,  BFG+00,  CGF+00, 
CN02], 

An  organization’s  business  analysis  supports  the  decision  to  establish  a  product  line 
approach  with  product  line  goals,  objectives,  strategies  and  the  development  of  a  Concept 
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of  Operations  (CONOPS).  The  product  line  CONOPS  [AIA93,  CFM+96,  Coh99,  CN02] 
establishes  an  approach  for  achieving  the  organization’s  product  line  goals  of  doing  a  job 
better,  faster,  and  cheaper  by  focusing  on  efforts  that  reduce  the  costs  and  risks,  associated 
with  system  development.  The  CONOPS  is  the  system-users  operational  view  of  the  sys¬ 
tem  under  development,  and  [Coh99]  notes  the  CONOPS  identifies  in-house  product  line 
responsibilities  for  the  government  and  commissioning  organizations,  and  establishes  ac¬ 
quisition/supplier  relationships,  detennines  the  ownership  of  product  line  assets  and  defines 
access  policies. 

However,  there  are  also  significant  implementation  issues  with  product  lines  identi¬ 
fied  by  [WL99,  and  EbeOl].  As  a  result,  the  SEI  explored  the  range  of  issues  and  practices 
necessary  for  the  successful  implementation  of  software  product  lines  [KZ96,  Cle97, 
CN02],  and  developed  several  products  and  including  an  investment  analysis  methodology 
for  software  product  lines  [Wit96].  The  SEI  also  sponsored  a  series  of  Product  Line  Prac¬ 
tice  (PLP)  Workshops  [BCC+97,  BCC+98,  BCN+98,  BCD+00,  CGF+00]  to  share  industry 
practices  in  software  product  lines  and  to  explore  the  technical  and  non-technical  issues 
involved  with  software  product  line  ventures,  including  software  engineering,  technical 
management,  and  enterprise  management  functions,  responsibilities,  and  issues. 

2.  Challenges  for  Software  Product  Lines 

a.  Developing  Software  Product  Lines  in  the  Department 

Any  software  product  line  implementation  strategy  in  the  Department  must 
take  into  account  the  Department  is  an  acquirer  of  software  systems  and  not  the  software 
developer,  and  factor  in  product  line  practice  areas  supporting  the  Department’s  acquisition 
process.  Initial  efforts  to  employ  product  lines  in  the  Department’s  M&S  domain  have 
been  limited.  The  Air  Force  employed  a  structural  modeling191  [ASC94,  CB96]  initiative 
to  support  the  development  of  aircrew  trainer  simulator192  software  for  the  B-2  Weapons 


Structural  modeling  has  been  in  use  since  the  mid-1980s  during  which  it  was  expanded  from  a  method  for  constructing  a  software 
architecture  into  an  architecture-based  development  method.  It  includes  the  general  engineering  principles  and  technologies  including 
prescriptive  architecture,  incremental  development,  and  prototyping  [CB96]. 

192 

A  simulator  is  a  device,  computer  program,  or  system  that  performs  simulation  or  for  training,  a  device  which  duplicates  the  essential 
features  of  a  task  situation  and  provides  for  direct  human  operation  [DoD98]. 
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System  Trainer  (1986),  C-17  Aircrew  Training  System  (ATS)  (1990),  Special  Operation 
Forces  (SOF)  ATS  (1991),  and  the  Simulator  Electric  Combat  Training  (1992)  as  a  method 
to  improve  changeability193  and  resolve  the  limitations194  of  the  previously  employed  data- 
driven  architecture  19\  [CB96]  noted  that  the  simulation  software  was  only  a  part  of  the  B2 
trainer  supporting  the  physical  cockpit,  the  cueing  system,  and  the  display  software,  and 
not  a  complete  architectural  solution,  as  presented  by  this  dissertation.  A  major  point  ad¬ 
dressed  by  [BC96]  is  that  non-technical  issues  including  business,  personnel,  management 
practices,  cultural  and  process  issues  are  equally  as  important  to  the  success  of  a  product 
line  initiative  as  the  technical  issues. 

Several  Department  research  and  acquisition  organizations  recently  explored 
product  lines  applicability  within  the  Department’s  M&S  domain.  The  Naval  Postgraduate 
School  (NPS)  developed  a  body  of  knowledge  germane  to  this  research  initiative  with  re¬ 
search  on  the  Janus  model  [JBL97,  SLB+98,  SLB+99,  BLS+99],  but  it  was  limited  in 
scope  to  a  single  model.  The  U.S.  Army  will  use  product  lines  as  a  method  to  replace  out¬ 
dated  legacy  instrumentation  and  simulation  systems  used  in  support  of  the  Maneuver 
Combat  Training  Centers  with  the  next  generation  Objective  Instrumentation  Systems 
[DDW01,  MD02],  However,  the  [DDW01,  MD02]  initiative  appears  more  focused  on  the 
hardware  components  of  a  product  line,  and  it  is  much  too  soon  too  soon  to  determine  if  it 
will  be  successful.  [Bru98]  also  evaluated  the  product  line  methodology  to  detennine  the 
feasibility  of  the  product  line  approach  for  software  development  in  a  single  model. 

The  Department  recognizes  the  potential  benefits  of  product  lines  along  with 
an  acknowledgment  of  the  significant  challenges  to  adopting  this  approach  [Wit96,  JNR98, 
BFG+00,  JonOO,  BOS02,  Cam02,  CDS02].  Many  of  the  challenges  stem  from  the  fact  that 
the  Department  is  in  the  business  of  acquiring  systems  rather  than  developing  them.  Al¬ 
though  product  line  practice  works  in  industry,  many  attempts  to  emulate  this  success 
within  the  Department  encountered  problems  [BNS97,  BFG+00].  Product  lines  face  im- 
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Changeability  is  defined  as  the  ease  with  which  the  software  standard  can  be  maintained  throughout  its  life  cycle,  and  extends  the 

concepts  of  modifiability  and  maintainability  to  include  changes  in  requirements  and  specifications  [CN96]. 

194 

The  data-driven  software  architecture,  a  de  facto  standard  since  the  1960s  increased  the  complexities  and  risk  during  integration  and 
maintenance  phases  of  the  life  cycle,  most  significantly  data  coherence  problems  stemming  from  spreading  out  the  state  information  and 
communication  of  the  state  information  across  subsystems  [CB96]. 

195 

The  data-driven  architecture  consists  of  the  executive  code/scheduler,  system  routines,  scheduling  table,  and  the  data  pool,  which  is 
the  data  area  shared  by  the  system  routines  for  storage  and  communication  of  state  information  [CB96] 
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plementations  in  the  Department  face  technical  challenges  and  significant  non-technical 
barriers  such  as  culture  and  acquisition-related  issues.  The  SEI  identified  several  key  is¬ 
sues  an  organization  should  address  before  moving  to  a  product  line  approach  for  acquiring 
or  developing  software: 

•  What  constitutes  the  product  line? 

•  How  will  it  be  introduced? 

•  What  key  organizational  elements  will  be  involved  in  defining,  developing,  and 
fielding  the  product  line? 

•  What  is  the  relationship  between  product  line  assets  and  systems  within  the  product 
line? 

•  How  will  the  architecture  be  developed  and  maintained? 

•  What  are  the  sources  of  software  components  [BNS97,  BFJ99,  BFG+00]? 

[BFJ99]  recommends  three  product  line  acquisition  activities  for  acquiring  a 
product  line  in  the  Department’s  acquisition  environment.  These  activities  include:  1)  ac¬ 
quire  architecture  and  other  elements  of  an  asset  base  to  enable  a  product  line  approach;  2) 
acquire  software  products  utilizing  the  asset  base;  and  3)  acquire  the  services  to  maintain 
the  asset  base  and  support  the  development  and  enhancement  of  derivative  products 
[BFJ99].  [BFJ99]  also  suggests  that  product  line  start-ups  may  be  more  successful  in  a  sys¬ 
tem’s  operational  support  phase,  vice  system  start  up  phase.  [BFJ99]  also  believes  the  stra¬ 
tegically  positioned  system  sustainers  may  be  motivated  to  adopt  a  product  line  approach, 
since  they  may  have  responsibility  for  sustaining  and  enhancing  similar  operational  sys¬ 
tems. 


b.  Component  Development  for  Software  Product  Lines 

Components196  play  a  major  role  in  many  software-intensive  systems 
[LG96,  HBL97,  BBB+00,  CN02,  FEA02b,  SAG02].  There  are  several  motivations  for  a 
components-based  software  approach  to  software:  providing  a  basis  for  commerce  in  reus¬ 
able  software  to  meet  demand,  facilitating  the  development  of  flexible  systems,  and  reduce 
the  time  to  design,  implement,  and  deploy  systems.  As  an  indication  of  the  need  for  soft- 
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A  software  component  is  an  opaque  implementation  of  functionality,  subject  to  third-party  composition,  conformant  with  a  compo¬ 
nent  model  [BBB+00]. 
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ware  components  at  the  United  States  Federal  Government  level,  the  Office  of  Manage¬ 
ment  and  Budget  recently  established  the  Federal  Enterprise  Architecture  Program  Man¬ 
agement  Office  to  remedy  the  lack  of  a  Federal  Enterprise  Architecture  (FEA)  [FEA02b, 
Hay03b],  a  major  barrier  to  the  success  of  the  E-Government  initiative  .  The  FEA  pro¬ 
gram  includes  a  Component-Based  Architecture  (CBA)  [FEA02b]  supported  by  tools, 
technologies,  and  standards  facilitating  component  reuse,  distribution,  and  cross-agency 
collaboration. 

The  CBA  framework  [FEA02b,  Gar02]  builds  on  a  tier-based  architecture, 
employing  layers  to  support  the  creation  of  components  facilitating  reuse  and  interoperabil¬ 
ity.  The  integrated  CBA  model  [FEA02b]  includes  five  architecture  layers:  Infonnation, 
Technical,  Security,  Application,  and  Business  employing  an  “Import  and  Export”  method¬ 
ology  supported  by  Extensible  Markup  Language  (XML)199  schemas  to  support  inter¬ 
agency  interoperability  and  data  sharing.  The  CBA  [FEA02b]  involves  relevant  industry 
standards  (e.g.,  Hypertext  Markup  Language  (HTML),  XML,  XML  Web  Services,  Simple 
Mail  Transfer  Protocol  (SMTP),  and  Simple  Network  Management  Protocol  (SNMP))  pro¬ 
viding  a  foundation  for  interoperability,  growth,  integration,  and  expansion. 

The  Department  supports  the  CBA  initiative  in  many  areas  [Lat97],  includ¬ 
ing  M&S.  In  the  Department’s  M&S  arena,  the  DMSO  initiated  the  Composable  Mission 
Space  Environment  (CMSE)200  project  and  sponsored  a  Composable201  M&S  Workshop 
[ES02,  Gar02,  Lor02,  Mat02,  Pet02,  Ves02,  ZHS02]  in  July  2002  to  explore  ways  compo¬ 
nent-based  technology  may  reduce  the  manpower,  time,  and  resources  currently  needed  to 
execute  Department  simulations  in  a  transformation  environment  with  joint,  interoperable, 
re-useable  models  [CA02,  PLN02,  RPK+02,  ZHS02],  The  Department’s  on-going  trans¬ 
formation  process  drives  the  transition  requirement  from  a  threat-based  force  applying  sys¬ 
tem-focused,  platfonn-centric  approaches  to  a  capabilities-based  force  employing  mission- 
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At  the  Government  Enterprise  level  a  component  is  an  application,  capability,  or  service  that  leverages  technology  to  perform  a  spe¬ 
cific  business  function  [FEA02b]. 
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The  President  approved  twenty- four  priority  E-Gov  initiatives  to  meet  the  business  needs  of  the  Federal  Government  [FEA02b]. 

199 

XML  is  a  platform  independent,  universal  language  used  to  support  the  structuring  and  integration  of  documents  and  data  on  the  web 
with  a  flexible  set  of  standards  for  tagging  and  classifying  information  readable  by  humans  and  data  exchange  systems  [FEA02b]. 

The  CMSE  is  an  interrelated  collection  of  enabling  M&S  technologies,  tools,  and  procedures. 

201 

Composability  is  the  capability  to  select  and  assemble  simulation  components  in  various  combinations  into  simulation  systems  to 
satisfy  specific  user  requirements  [Pet02]. 
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focused  network-centric  approaches  in  how  the  Department  trains,  procures,  equips,  oper¬ 
ates  and  fights.  The  Workshop  identified  several  specific  recommendations  for  compos- 
able  M&S  standard  development: 

•  Develop  and  interchange  standard  for  data  and  models, 

•  Develop  a  Composable  M&S  Ontology, 

•  Conduct  a  business  case  analysis, 

•  Conduct  further  research  in  many  less-understood  areas  including  the  impact  of  cul¬ 
ture  and  organizations  on  adoption  of  a  composable  M&S  approach, 

•  Engage  basic  research  in  component  technology  needs, 

•  Identify  VV&A  issues  [ZHS02]. 

The  SEI  recently  initiated  a  component-based  software  engineering  (CBSE) 
study  to  detennine  if  they  could  extract  predicted  properties  of  a  CBSE  system  of  compo¬ 
nents  made  from  the  components  themselves  [BBB+00].  During  the  process  [BBB+00] 
adopted  the  following  narrow  vision  of  CBSE: 

Component-based  software  engineering  is  concerned  with  the  rapid  assem¬ 
bly  of  systems  from  components  where  components  and  frameworks  have 
certified  properties',  and  these  certified  properties  provide  the  basis  for  pre¬ 
dicting  the  properties  of  systems  built  from  components  [BBB+00] . 

Component-based  systems  rely  on  defined  standards  and  conventions,  or 
component  model,  and  a  support  infrastructure,  or  component  framework  [BBB+00].  A 
component  model  specifies  the  component  standards  and  conventions  (e.g.,  component  typ- 
ing,  interaction  schemes,  resource  binding"  )  whereas  a  component  framework  provides 
the  developer  with  an  implementation  of  services  supporting  or  enforcing  component 
model  standards  and  conventions  [BBB+00] .  Although  there  still  remains  a  lack  of  con¬ 
sensus  on  the  contents  of  a  component  model,  [BBB+00]  suggest: 

•  A  uniform  composition  constraining  how  components  interact  if  and  only  if  they 
share  consistent  assumptions  about  what  each  component  provides  and  requires  of 
another  component.  Although  some  assumptions  are  unique  to  the  component,  a 
standardization  of  assumptions  (e.g.,  component  location,  control  flow,  data  encod¬ 
ing,  communication  protocol)  reduces  chances  for  accidental  mismatch,  which  ad¬ 
versely  affect  composition. 


202  A  component  type  is  defined  by  the  interfaces  it  implements.  An  interaction  scheme  specifies  how  components  are  located,  commu¬ 
nication  protocols,  and  quality  of  service  attributes  including  security  and  standard  transaction  assumptions.  Resource  binding  identifies 
the  process  of  composing  components  in  terms  of  binding  a  component  to  one  or  more  resources  [BBB+00]. 
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•  Appropriate  quality  attributes,  which  depend  on  the  software  architecture  and  archi¬ 
tectural  style203  supports  the  desired  quality  attributes  for  a  system  composed  from 
third-party  components  and  the  quality  of  service204. 

•  Deployment  of  components,  application  development,  and  the  emergence  of  a  vi¬ 
able  market  in  third-party  components  are  critical  to  the  success  of  a  CBSE  envi¬ 
ronment.  Historical  examples  abound  where  technologies  survived  and  economi¬ 
cally  prospered  in  the  face  of  arguably  superior  technology  (e.g.,  Apple  versus  Mi¬ 
crosoft,  the  Beta  tape  format  versus  the  VHS  tape  fonnat).  This  precondition  for 
component  composition  suggests  that  components  successfully  transition  from  the 
component  developer  to  an  application  developer’s  composition  environment,  and 
finally,  operate  in  the  customer’s  environment  [BBB+00]. 

The  second  component  concept  advanced  by  [BBB+00]  is  the  component 
framework.  In  an  analogous  sense,  a  component  framework  is  comparable  to  an  operating 
system,  in  which  components  are  to  frameworks  what  processes  are  to  operating  systems, 
and  the  framework  manages  resources  shared  by  the  components,  and  provide  the  underly¬ 
ing  services  enabling  communication  among  components  [BBB+00]. 

In  practice,  component  frameworks  include  the  Enterprise  JavaBeans™ 
specification  of  servers  and  containers,  the  WaterBeans  framework  for  real-time  visualiza¬ 
tion  of  high-performance  data  streams,  and  Microsoft’s  VisualBasic  framework  for  visual 
composition  of  components  [BBB+00].  A  major  challenge  facing  the  successful  imple¬ 
mentation  of  component  frameworks  is  the  issue  of  standard  versus  custom  component 
models  and  frameworks.  These  requirements  create  competition  between  the  maximum 
flexibility  school  of  thought  supporting  different  architectures,  different  styles,  and  differ¬ 
ent  allocation  of  quality  attributes  based  on  the  application;  and  the  business  case  school  of 
thought  supporting  the  establishment  of  a  viable  market  in  software  components 
[BBB+00]. 

Component  interfaces  are  the  foundation  for  component  substitutability  and 
significantly  more  complex  than  traditional  system  interfaces.  This  generated  the  idea  of 
an  interface  contract  [BBB+00]  where, 

•  A  contract  binds  two  or  more  parties, 

•  The  parties  negotiate  details  of  the  contract  before  they  sign  the  contract, 


203  The  architectural  style  includes  standardization  of  the  types  of  components  used  in  a  system  and  their  patterns  of  interaction 

[BBB+00]. 

204 

The  quality  of  service  specifies  the  patterns  of  interaction  (e.g.,  Type  of  transactions,  encryption  methods)  [BBB+00]. 
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•  The  contract  prescribes  normative  and  measurable  behavior  by  all  parties, 

•  A  contract  cannot  be  changed  unless  signatories  approve  all  changes  [BBB+00] . 

[LG96,  OHR99,  BCSOO,  PPOO,  GriOO,  ABMOO,  and  AB02]  explain  the  de¬ 
sign  of  generic  systems  based  on  the  reuse  of  software  designs  and  components,  supporting 
designs  of  families  of  systems  or  product  lines  based  on  commonalities,  including  compo¬ 
nent  metadata  [OHR99],  within  the  product  lines.  In  object-oriented  design,  a  similar  con¬ 
cept  for  extensible  software  system  provides  for  a  framework  and  specific  plug-ins.  Design 
quality  is  critical  and  some  quality  attributes  may  or  may  not  be  discemable  at  run-time, 
while  other  attributes  relate  to  the  architectural  qualities  such  as  conceptual  integrity,  cor¬ 
rectness,  completeness,  and  build  ability.  Four  general  principles  identified  by  [BL91, 
Som95,  BW97,  Pre97]  impact  software  component  construction: 

•  Reduction  of  complexity, 

•  Anticipation  of  diversity, 

•  Structuring  for  validation, 

•  Use  of  external  standards  [BL91,  Som95,  BW97,  Pre97]. 

Composition”  is  a  key  process  within  the  software  component  develop- 
ment  life  cycle.  The  software  composition  design"  activity  includes  analysis  of  the  soft¬ 
ware  requirements  [HBL97,  BocOO,  FHR00]  and  results  in  the  development  of  the  software 
product  [BL91,  Som95,  LG96,  LZB+96,  Lat97,  Pre97,  Jac98,  BBB+00,  RPG00,  CooOlb, 
NS01,  AKN02].  Key  enabling  design  techniques  and  principles  include  abstraction,  cou¬ 
pling,  cohesion,  decomposition,  modularization,  encapsulation,  information  hiding,  separa¬ 
tion  of  interface  and  implementation,  sufficiency,  completeness,  and  primitiveness. 

[Wil96,  VT01]  discuss  a  number  of  key  issues  for  consideration  during  the  design  phase 
including  quality  and  the  decomposition,  organization,  and  packaging  of  the  software  com¬ 
ponents.  [BW97,  CA02,  PLN02,  RPK02]  review  other  aspects  of  the  systems  behavior, 
which  horizontally  crosscut  the  system  functionality,  including  concurrency,  event  han¬ 
dling  and  control,  distribution,  exception  handling,  interactive  systems,  and  data  persis¬ 
tence  issues. 


Composition  is  the  term  used  in  component-based  development  to  explain  how  so  develop  or  assemble  systems  [BBB+00] 
[IEE90]  defines  software  design  as  both  a  process  and  a  product. 
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Several  critical  aspect  of  component-based  software  still  require  attention 
according  to  [BBB+00,  VT01,  and  Voa02],  including  the  need  for  systems  that  will  pre¬ 
dictably  exhibit  the  quality  attributes  required  of  them  (e.g.,  predictive  composition)  and 
address, 

•  Changeable  environment  and  dependencies, 

•  Variable  usage, 

•  Large  number  of  small  parts  to  manage, 

•  Component  flux  (e.g.,  version  creep), 

•  Developer  inconsistencies  [VT01], 

Key  technical  areas  of  component-based  development  addressed  earlier  by 
[LG96,  HBL97]  and  recently  by  [BBB+00]  include  components,  interfaces,  component 
models,  component  framework  [Lat97,  WilOl],  composition  [LG96,  WilOl,  AKN02, 
Voa02,  KH02,  PLN02],  component-based  product  line  engineering  [ABB+02],  the  compo¬ 
nent  certification  process  [WR94,  POPOO,  CSS+01,  GM01,  GS01,  JDL01,  MasOl,  MCOlb, 
PSR+01,  SW01],  and  component  wrapping  [PSR+01].  In  essence  at  this  point  on  the  ma¬ 
turity  curve,  the  current  body  of  knowledge  on  components,  component  trust  and  certifica¬ 
tion,  component  technology,  and  software  architecture, 

•  Raises  concerns  about  properties  of  assemblies  of  components, 

•  Lacks  information  about  component  behavior, 

•  Lacks  an  understanding  of  the  functional  and  extra-functional  properties  of  systems, 

•  Lacks  the  ability  to  determine  properties  of  “black-box”  component  assemblies 
[CSS+01], 

Currently  there  is  not  an  established  basis  for  how  well  component  models 
and  frameworks  contribute  to  achieving  the  desired  quality  [WR94,  LG96,  JDL01,  VT01, 
KH02,  AKN02,  Voa02];  nor  is  there  any  basis  for  assessing  the  quality  of  software  compo¬ 
nent  interfaces,  composition,  and  component  certification.  [WR94,  BBB+00,  KH02,  and 
Wr02]  address  components  and  component  certification  issues  and  analyzed  component 
complexity  and  interfaces.  [CTW98]  evaluated  software  component  licensing  issues,  while 
[Voa02]  reviews  several  significant  issues  with  composition  practices.  Consensus  on  the 
many  solutions  required  for  contentious  component  /  composition  issues  do  not  appear  on 
the  near-term  horizon. 
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3. 


Product  Line  Practice  Areas 


[CN02,  Nor03]  defines  a  software  product  line  practice  area  as  a  body  of  work  or  a 
collection  of  activities,  which  an  organization  must  successfully  master  to  carry  out  the  es¬ 
sential  work  of  a  software  product  line.  Many,  if  not  the  majority  of  the  practice  areas  de¬ 
scribe  activities,  which  are  critical  to  the  success  of  any  software  project,  and  provide  start¬ 
ing  points  for  organizations  to  initiate  and  master  activities  and  measure  progress  in  adopt¬ 
ing  a  product  line  approach.  The  practice  areas  in  the  framework  divide  into  three  catego¬ 
ries:  1)  software  engineering  practices,  2)  technical  management  practices,  and  3)  organiza¬ 
tional  management  practices  illustrated  in  Table  8-1. 

Implementing  software  product  lines  offers  potential  gains,  and  entails  significant 
risks,  including  technical,  organizational,  and  management  issues.  [Nor03]  explains  that 
building  and  acquiring  a  software  product  line  requires  disciplined  engineering  supported 
by  mature  technical  and  organization  management  processes  of  universal  essential  activi- 
ties  and  practices,  which  the  SEI  developed  into  an  online  framework  ,  a  web-based 
document  describing  the  competencies  needed  to  develop  and  field  a  successful  a  software 
product  line  or  any  software-intensive  system. 

A  Framework  for  Software  Product  Line  Practice,  (Version  3.0)  [Nor03]  is  a  web- 
based  tool  introduced  and  designed  to  support  the  software  community  in  software  product 
line  endeavors.  Each  version  represents  an  incremental  attempt  to  capture  information 
about  successful  software  product  line  practices.  This  infonnation  builds  on  the  work  of 
the  SEI  Product  Line  Practice  Initiative,  including  studies  of  organizations,  which  have 
built  product  lines,  from  direct  collaborations  on  software  product  lines  with  customer  or¬ 
ganizations,  and  from  leading  practitioners  in  software  product  lines.  The  SEI  website  de¬ 
fines  all  of  the  practice  based  upon  ongoing  software  product  line  collaborations  and  the 
feedback  from  the  community.  Future  versions  will  build  upon  the  current  foundation  and 
the  growing  body  of  knowledge,  refining  current  knowledge,  and  describing  a  small  num- 
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Framework  in  this  context  suggests  a  conceptual  index,  a  frame  of  reference  for  the  information  essential  for  success  with  product 
lines  [Nor03]. 
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ber  of  product  line  scenarios  involving  the  development,  acquisition,  and/or  evolution  of  a 
software  product  line.  The  Product  Line  Practice  envisions: 

•  Product  line  development  as  low  risk  /  high  return 

•  Techniques  for  exploiting  system  commonalities  and  controlling  variability  with 
standard  Department,  government,  and  industry  software  engineering  practices 
[Nor03]. 

There  are  several  overlapping  tenns  from  previous  information  technology  areas 
synonymous  with  product  line  terms.  [ADD+00,  TPOO,  and  SchOOa]  address  core  asset  de¬ 
velopment  activities,  (e.g.,  domain  engineering),  while  [BFG+00]  addresses  product  devel¬ 
opment  from  core  assets,  often  cited  as  application  engineering.  The  architecture  and 
components  are  central  to  the  set  of  core  assets  (e.g.,  platform),  used  to  construct  and 
evolve  the  products  in  the  product  line.  Development  is  a  generic  term  to  describe 
ac q u i r i riEhsaufi ware c v c ra  1  viable  options  for  acquiring  software  [CN02,  Nor03,  MNJ02],  An 
organization  may  build  the  software  in-house  from  scratch,  mine  existing  legacy  software 
assets  (e.g.,  core  assets),  purchase  commercial-off-the-shelf-software  (COTS),  or  commis¬ 
sion  the  development,  normally  through  a  contract  with  someone  to  build  the  software 
[Nor03,  MNJ02],  or  even  combine  several  options.  The  objective  of  the  core  asset  activity 
is  to  support  development  of  a  production  capability  for  products  and  requires  three  com¬ 
ponent  activities: 

•  Defining  the  product  line  scope, 

•  Developing  the  production  plan 

•  Producing  the  core  assets,  [Nor03], 

Table  8-1  identifies  practice  areas  as  a  body  of  work  or  a  collection  of  activities, 
which  an  organization  must  successfully  master  to  carry  out  the  essential  work  of  a  soft¬ 
ware  product  line.  The  following  sections  describe  the  three  categories  and  the  associated 
practice  areas.  The  number  of  products  developed  from  core  assets  will  form  the  founda¬ 
tion  for  metrics  indicating  the  real  potential  for  mining  core  assets  [BFG+00,  BOSOO].  A 
short  summary  of  each  product  line  practice  area  follows. 
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PRODUCT  LINE  PRACTICE  AREAS  v4.i 

Software  Engineering 

Technical  Management 

Organization  Management 

•Architecture  Definition 

•Configuration  Management 

•Business  Case 

•Architecture  Evaluation 

•Process  Definition 

•Customer  Interface  Mgt 

•Component  Devlopment 

•Product  Line  Scoping 

•Acquisition  Strategy 

•COTS  Utilization 

•Technical  Planning 

•Funding 

•Mining  Existing  Assets 

•Technical  Risk 

•Launching/Institutionalize 

•Requirements  Engineering 

Management 

•Market  Analysis 

•Software  System 

•Tool  Support 

•Operations 

Integration 

•Data  Collection,  Metrics, 

•Org  Planning 

•Testing 

Track 

•Org  Risk  Mgt 

•Understanding  Relevant 

•Make/Buy/Mine/ 

•Structure  Org 

Domains 

Commission 

•Org  Technology  Forecast 

•Training 

Table  8-1.  Product  Line  Practice  Areas  (From  [Nor03]) 
a.  Software  Engineering  Practice  Area 

Based  on  research  to  date,  the  SEI  [CN02,  Nor03]  proposes  the  following 
software  engineering  practices  listed  in  Table  8-1  as  necessary  practices  supporting  an  or¬ 
ganization’s  capability  and  technology  to  create,  evolve,  and  maintain  core  assets  and 
products: 

•  Architecture  Definition  -  This  practice  area  defines  the  activities  supporting  the 
definition  of  software  architecture  for  a  product  line  and  addresses  quality  attrib¬ 
utes,  system  interaction  requirements,  and  organization  business  goals.  The  focus 
may  be  infrastructure  focused  (e.g.,  operating  system,  protocols,  middleware),  or 
focus  on  the  application  functionality.  The  software  architecture"  is  a  major  com¬ 
ponent  supporting  the  development  of  the  core  assets,  affecting  how  well  an  organi¬ 
zation  fields  products  form  a  shared  asset  repository,  supporting  explicitly  allowed 
variations  [CN02,  Nor03]. 

•  Architecture  Evaluation  -The  architecture  of  a  system,  an  early  design  artifact, 
represents  one  of  the  earliest  design  decisions,  and  perhaps  the  most  difficult  to 


208  No  completely  accepted  definition  for  software  architecture  has  emerged,  although  over  ninety  have  been  discovered  by  [BCC+03]. 
[CBB+03]:  suggests  the  following  definition:-  Software  Architecture  is  the  structure  or  structures  of  a  system,  which  comprise  elements, 
their  externally  visible  properties,  and  the  relationships  among  them  [CBB+03]. 
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change  over  time.  It  is  at  this  point  that  the  software  architects  establish  the  sys¬ 
tem’s  quality  foundations  (e.g.,  security,  reliability,  usability).  The  abstraction  of 
the  architecture  at  this  point  supports  communication  among  the  system’s  key 
stakeholders.  Several  different  architecture  evaluation  techniques  exist  including: 
the  Architecture  Tradeoff  Analysis  Method,  the  Software  Architecture  Analysis 
Method,  active  design  reviews,  Software  Perfonnance  Engineering,  and  active  re¬ 
views  for  intermediate  designs  [CN02,  Nor03], 

•  Component  Development  -  The  term  component209  like  software  architecture  and 
object  has  collected  many  meanings,  and  not  yet  standardized.  The  component 
functionality  is  normally  encapsulated  and  packaged  with  an  interface  provided  to 
other  components  employing  an  agreed  upon  interconnection  method.  Components 
form  the  building  blocks  for  applications,  linked  by  communication  and  connection 
technologies  including  the  Common  Object  Request  Broker  Architecture 
(CORBA),  from  the  Object  Management  Group;  Distributed  Common  Object 
Model  (DCOM),  originally  from  Microsoft;  and  JavaBean  [DRS99],  developed  by 
Sun  [CN02,  Nor03]. 

•  COTS  Utilization  -  Commercial-off-the-shelf  and  non-developmental  items  (NDI) 
may  also  be  used  as  core  assets  in  a  product  line.  In  the  past  decade,  the  growth  of 
middleware  technologies  supported  the  employment  of  COTS  /  NDI  components  in 
large-scale  systems,  introducing  a  new  set  of  risks,  constraints,  and  tradeoffs 
[Nor03].  Key  issues  for  consideration  include  COTS  evaluation  process,  adaptabil¬ 
ity,  and  vendor  support  for  COTS  products  [BB99,  Car99,  Voa99,  HP99b, 
SWR+99,  PHW99,  MGOO,  ABCOO,  CN02], 

•  Mining  Existing  Assets  -  Much  of  today’s  software  systems  evolve  as  extensions 
of  legacy  systems.  However,  legacy  software  involves  resurrected  and  rehabili¬ 
tated  software  or  service  in  a  new  system  beyond  the  original  design.  Mining  ex¬ 
pands  significantly  upon  the  current  practice  of  small-grained  reuse  of  code,  sub¬ 
routines,  or  small  programs  and  may  include  a  higher-level  focus  on  the  organiza¬ 
tion’s  business  processes  and  software  architecture,  in  addition  to  consideration  of 
the  normal  constraints  of  cost,  schedule  and  functionality.  Mining  assets  are  re¬ 
source  intensive  activities  and  focus  on  a  wide  range  of  assets  besides  code,  includ¬ 
ing  business  models,  rule  bases,  and  budgets.  Prime  candidates  include  algorithms, 
interface  specifications,  perfonnance  models,  test  plans,  and  available  architecture 
documentation.  If  quality  documentation  does  not  exist,  reconstruction  of  existing 
documentation  with  enhancements  such  as  tradeoff  options,  presents  an  alternative 
approach  supporting  improved  use  of  the  component  [BOSOO,  CN02,  Nor03]. 

•  Requirements  Engineering  -  The  IEEE  defines  a  requirement210,  and  [Bro87] 
noted  the  “hardest  single  part  of  building  a  software  system  is  deciding  precisely 
what  to  build. .  .the  detailed  technical  requirements”  [Bro87].  [BL91,  GGR+94, 
GGR+95,  Som95,  Pre97,  LKOO]  provide  in-depth  coverage  of  requirements  engi- 


9  Components  are  the  principal  computational  elements  and  data  soures  that  execute  in  a  system  [CBB+03]. 

210 

A  requirement  is:  (1)  A  condition  or  capability  needed  by  the  user  to  solve  a  problem  or  achieve  an  objective.  (2)  A  condition  or 
capability  that  must  be  met  or  possessed  by  a  system  or  a  system  component  to  satisfy  a  contract,  standard,  specification,  or  other  for¬ 
mally  imposed  documents.  (3)  A  documented  representation  of  a  condition  or  capability  as  in  (1)  or  (2)  [IEE90]. 
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neering  techniques  supporting  systematic  and  repeatable  processes  for  completely, 
consistently,  and  relevantly  eliciting,  analyzing,  specifying,  verifying,  and  manag¬ 
ing  requirements.  Requirements  engineering  for  a  product  line  differs  from  the  re¬ 
quirement  process  of  a  single  system  and  includes:  capturing  the  anticipated  varia¬ 
tions  over  the  projected  future  of  the  product  line,  finding  commonalities  and  identi¬ 
fying  variations,  preparing  a  product-line-wide  set  of  requirements  and  product  - 
specific  requirements.  It  may  involve  a  broader  verification  process  occurring  most 
likely  in  stages,  and  adaptation  to  support  the  dual  nature  and  staged  (e.g.,  common, 
specific)  nature  of  product  line  requirements  engineering  [CN02,  Nor03]. 

Product  line  analysis  is  a  requirements  process  for  engineering  a  product 
line  of  software-intensive  systems  [CDK+01].  The  [CDK+01]  suggested  method¬ 
ology  encompasses  the  elicitation,  analysis,  specification,  and  verification  and  vali¬ 
dation  of  the  requirements  for  a  product  line.  Four  interrelated  products  support  the 
product  line  requirements  model  for  elicitation,  analysis,  specification,  and  verifica¬ 
tion:  object  modeling  [RF01],  use-case  modeling,  feature-modeling,  and  the  dic¬ 
tionary,  with  the  goal  to  achieve  high  cohesion  /  low  coupling  [CDK+01], 
[CDK+01]  also  propose  product  line  analysis  methods  for  establishing  the  require¬ 
ments  for  a  product  line  of  software-intensive  systems  supported  in  the  context  of 
product  line  development,  assisted  by  modeling  techniques  and  tools  [YMK+00, 
SSP+00,  STOO,  HOF+OO,  CN02]. 

[CDK+01]  also  show  how  to  build  a  requirements  model  from  work  prod¬ 
ucts,  based  on  object  modeling,  use-case  modeling,  and  feature-modeling  tech¬ 
niques.  Requirements  are  statements  of  what  a  system  must  do,  how  the  system 
must  behave,  the  properties  it  must  exhibit,  the  qualities  it  must  possess,  and  the 
constraints  that  the  system  and  its  development  must  satisfy.  A  feature  is  a  distinct 
aspect,  quality,  or  characteristic  of  a  software  system  or  systems  visible  to  the  user. 
Strategies  and  methods  for  product  line  requirements  modeling  [KNOO,  CJBOO, 
AOV+OO]  include  the  feature-driven  strategy  and  a  use-case-driven  strategy.  In  a 
feature-driven  approach,  requirement  modeling  focuses  on  the  features,  while  de¬ 
velopers  use  the  use-case  strategy  to  discover  requirements  [CDK+01], 

•  Software  System  Integration  -  This  practice  involves  combining  individual  soft¬ 
ware  components  into  an  integrated  whole  in  a  process  where  components  are  inte¬ 
grated  into  subsystems  or  when  subsystems  are  combined  into  products,  either  dis¬ 
cretely  supporting  a  waterfall  approach,  or  continuously  supporting  an  incremental 
methodology.  A  key  point  about  product  line  integration  is  that  cost  of  integration 
identified  in  the  product  line  scope,  core  assets,  and  production  plan  for  the  archi¬ 
tecture’s  planned  life  cycle  is  amortized  over  many  products,  versus  a  single  system 
integration.  Once  the  components  and  interfaces  have  been  tested  and  verified  there 
should  be  very  little  effort  needed  for  new  variations  and  adaptations.  Specific 
practices  supporting  component  integration  includes  patterns  [Fow97],  object  tech¬ 
nology,  wrapping  for  recovery  or  discovery  solutions,  and  middleware  [CN02, 
Nor03]. 

•  Testing  -  Testing  performs  two  primary  functions  described  in  detailed  by  [BL91, 
Som95,  Pre97,  Roa98,  and  LKOO].  First,  testing  continually  supports  developers 
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ability  to  identify  faults  leading  to  failure  in  the  development  phase,  and  secondly, 
at  a  later  time  when  testers  detennine  whether  a  system  can  perform  to  meet  its  re¬ 
quirements  [CN02,  Nor03].  Testing  is  also  a  critical  part  of  the  V&V  and  quality 
processes.  Metrics  and  measurements  for  testing  and  reliability  supporting  credibil¬ 
ity  in  simulation  software  include  [Ber94,  CFG+94,  Jon95,  FV96,  PGF96,  Sta96, 
MV96,  JA97,  Wie97b,  AVL+97,  Sch99b]. 

•  Understanding  Relevant  Domains  -  Domain  understanding  evolved  as  a  major  fac¬ 
tor  in  this  research.  Domains  are  areas  of  expertise  and  domain  knowledge  avail¬ 
able  for  creating  future  systems  or  set  of  systems,  with  an  understanding  that 
knowledge  from  several  domains  is  normally  required  to  build  a  single  product. 
Understanding  a  specific  domain  normally  entails  the  identification  of  areas  of  ex¬ 
pertise,  identification  of  recurring  domain  problems  and  known  solutions,  capturing 
and  representing  this  information  to  stakeholders,  for  the  duration  of  the  effort. 
Domain  comprehension  supports  understanding  the  commonality  and  variability  of 
potential  future  product  within  the  scope  of  the  product  line  [CN02,  Nor03]. 

b.  Technical  Management  Practice  Area 

Technical  management  practices  are  those  management  practices  necessary 
to  engineer,  develop,  evolve,  and  maintain  to  core  assets  and  the  products  and  encompass: 


•  Configuration  Management  -  Software  configuration  management  (CM)  [Bro98, 
CN02,  Nor03,  BCC+Olb]  includes  the  following  activities:  software  configuration 
identification,  software  configuration  control,  software  configuration  status  ac¬ 
counting,  software  configuration  auditing,  and  software  release.  Configuration 
management  processes  matured  steadily  since  the  early  1970s,  and  include  new  or 
reengineered  methods,  management  techniques  [Bro96,  BL91,  Som95,  Pre97, 
LK00],  technology  [STS94a],  and  products  [BH99]  continuously  evolve  [Bro87]. 

However,  CM  for  a  product  line  is  complex.  The  core  assets  and  each  of  the 
products  in  the  product  line  constitute  a  configuration  to  manage,  and  the  manage¬ 
ment  of  all  configurations  needs  coordination  under  one  process.  We  will  compare 
and  contrast  the  CM  requirements  for  a  single  system  with  the  CM  requirements  for 
a  product  line.  First,  a  single  system  manages  each  version’s  configuration;  a  prod¬ 
uct  line  requires  CM  for  each  version  of  each  product.  A  single-system  CM  process 
may  separately  manage  each  product  and  all  its  versions;  however,  in  a  product  line 
system  the  core  assets  integrated  across  all  products  require  a  single,  unified  CM 
process.  Third,  in  a  single  system,  the  component  developers  and  product  develop¬ 
ers  are  often  the  same,  whereas  in  a  product  line  must  support  the  CM  of  the  core 
assets,  normally  developed  by  one  team  and  supporting  parallel  production  by  sev¬ 
eral  other  teams  [CN02,  Nor03].  The  additional  requirements  levied  by  a  product 
line  CM  suggest  the  need  for  disciplined  CM  techniques  and  processes,  supported 
by  robust  CM  tools. 
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Additional  new  CM  techniques  include  the  uniform  version  management 
framework,  explained  by  [WMC02],  which  supports  the  definition  of  version  mod¬ 
els  and  addresses  the  orthogonal  differences  between  the  version  and  data  models;  a 
test  bed  model  that  separates  CM  repositories  from  CM  policies  detailed  by 
[HCH+02];  visualization  techniques  [EGF02],  and  software  merging  version  con¬ 
trol  [Men02].  [AB02]  describes  open  source  software  projects  CM  methods,  while 
[SJW+02]  discusses  open  source  software  maintainability.  [Kil02]  cites  the  follow¬ 
ing  benefits  to  software  accuracy  attributable  to  good  software  configuration  man¬ 
agement:  longer  shelf  life  of  V&V  work,  confidence  that  model  results  are  consis¬ 
tent,  lower  likelihood  of  undetected  errors  in  the  code,  less  experimentation  and 
training  needed  to  operate  the  model,  and  CM  provides  a  venue  to  pool  resources 
for  model  changes,  including  legacy  models  with  multiple  users  [Kil02], 

Configuration  management  has  proven  inconsistent  in  the  Department’s 
M&S  domain.  Citing  the  restrictions  of  the  current  VV&A  directives,  [Cau95]  is 
concerned  that  the  process  is  so  complex,  time-consuming  and  expensive  that 
changes  are  often  made  to  the  M&S  before  the  V&V  process  is  completed,  poten¬ 
tially  invalidating  the  process.  [Mue97a]  described  the  Susceptibility  Model  As¬ 
sessment  and  Range  Test  (SMART)  project,  which  integrated  configuration  man¬ 
agement  with  V&V  processes  to  improve  M&S  credibility.  Decision-makers  trust 
in  an  M&S  tool  requires  strict  configuration  management,  yet  the  M&S  analytical 
tool  set  must  have  the  flexibility  and  depth  to  resolve  complex  problems. 

•  Process  Definition  -  The  process  definition  practice  area  involves  an  organization’s 
capability  to  define  and  follow  documented  processes.  Product  line  management 
requires  the  disciplined  interaction  of  separate  organizational  entities  adhering  to 
mature  processes.  Each  core  asset  has  an  attached  process  explaining  how  to  em¬ 
ploy  the  core  asset  to  build  products  in  a  product  line  [CN02,  Nor03].  Software 
process  modeling  supports  process  definition,  describing  the  abstract  description 
and  models  of  important  defined  software  development  processes  executed  by  a 
human  or  a  machine  to  achieve  the  following  goals:  1)  facilitate  human  understand¬ 
ing,  2)  support  process  management,  and  3)  support  process  improvement.. 

•  Product  Line  Scoping  -  Scoping  bounds  a  system  or  set  of  systems  defining  behav¬ 
iors,  characteristics  or  aspects  included  or  excluded  from  the  product  line  formal¬ 
ized  in  a  scope  definition  document  supporting  the  requirements  engineering  proc¬ 
ess,  or  influencing  the  market  analysis  for  a  product  line  variant  [CN02,  Nor03]. 

•  Technical  Planning  -  The  product  line  requires  no  new  planning  processes,  how¬ 
ever,  core  asset  development,  core  asset  maintenance,  core  asset  production,  and 
core  asset  reuse  plans  are  unique  product  line  endeavors  [CN02,  Nor03].. 

•  Technical  Risk  Management  -  Risk  management  is  the  practice  of  managing  risks 
within  a  project,  organization  or  a  group  of  organizations.  Risk  management  is  a 
critical  process  for  a  product  line  since  the  risks  involve  more  than  one  product, 
with  potentially  far-reaching  consequences  [CN02,  Nor03], 

•  Tool  Support  -  A  product  line  requires  tools  to  support  concurrent  development, 
employment  and  maintenance  of  multiple  core  asset  artifacts.  Most  likely  several 
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interoperable  computer-aided  software  engineering  tools  will  manage  the  product 
line  [CN02,  Nor03]. 

•  Data  Collection,  Metrics,  and  Tracking  -  Measurements  and  metrics  support  and 
guide  management  decisions  about  whether  organization  goals  and  objectives  are 
met  over  time  [CN02,  Nor03].  Product  line  systems  are  managed  in  many  ways 
just  like  a  single  product  system,  except  for  the  additional  requirements  to  track  the 
core  asset  development,  product  development,  and  overall  management  of  the 
product  line.  [Ber94,  CFG+94,  Jon95,  FV96,  PGF96,  Sta96,  MV96,  JA97, 

Wie97b,  AVL+97,  Sch99b,  and  IEE001]  provide  guidance  on  measurement  and 
metrics. 

•  Make  / Buy  /  Mine  /Commission  Analysis  -  Normally  a  product  may  be  built  in- 
house,  purchased  from  a  commercial  company,  commissioned  for  development,  or 
mined  from  in-house  assets  based  on  a  core  asset  development  decision  process  in¬ 
cluding  quality,  cost,  product  line  requirements,  architecture,  variation  flexibility, 
maintainability,  and  schedule  [BSW+99,  CN02,  Nor03].  The  product  development 
activity  depends  upon  the  product  line  scope,  core  assets,  production  plan,  and  the 
requirements  for  the  individual  products.  Product  line  organizations  are  flexible 
and  may  include  a  product  group  for  several  products,  or  one  product  group  per 
product.  Management  according  to  [WapOO,  VWOO,  TCOOO]  also  makes  assets 
available  for  reuse,  retains  responsibility  for  success  or  failure,  manages  external  in¬ 
terfaces,  creates  an  adoption  plan,  and  acts  as  or  empowers  the  product  line  cham¬ 
pion.  Management  support  required  changes  to  cultural  perspectives  [Wie96b]  and 
allows  new  products  to  align  with  existing  assets,  or  update  the  assets  to  meet  the 
new  requirements. 

• 

c.  Organizational  Management  Practice  Area 

Organizational  management  practices  are  those  management  practices  nec¬ 
essary  to  orchestrate  the  entire  core  assets  and  products  line  effort  and  include: 


•  Building  and  Communicating  a  Business  Case  -  There  are  several  generally  ac¬ 
cepted  costing  approaches  for  building  and  communicating  a  business  case,  includ¬ 
ing  top-down,  bottoms-up,  algorithmic,  analogy,  and  expert  judgment  [BBW+00, 
CN02].  A  significant  body  of  work  exists  on  cost  estimation  approaches  based  on 
project  characteristics.  Software  estimation  tools  include  the  non-proprietary  Con¬ 
structive  Cost  Model  models  first  introduced  by  Barry  Boehm  COCOMO  model  in 
1981,  the  Revised  Intermediate  COCOMO  (REVIC)  model,  Putnam’s  Software 
Lifecycle  Model  (SLIM)  and  Boehm’s  COCOMO  II;  in  addition  to  software  size 
estimation  tools  such  as  SEER-SSM  discussed  and  compared  by  [BNT93,  STS93a, 
STS93b,  STS94b,  McG97,  BA99,  and  NogOO].  [SK02]  advocates  the  development 
of  newer  empirical  cost  and  schedule  estimation  approaches  based  on  the  increased 
use  of  COTS,  application  generators,  simulations,  and  fourth  generation  languages 
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today,  while  [Wit96]  proposes  an  investment  analysis  tool  for  software  product  line 
business  case  development. 

•  Customer  Interface  Management  -  Managing  the  customer  interface  for  a  product 
line  differs  significantly  from  a  single-product  organization,  requiring  new  methods 
for  managing  customer  expectations,  negotiating  requirements,  developing,  evolv¬ 
ing,  and  maintaining  customer  products  and  providing  customer  support  [CN002, 
Nor03]. 

•  Developing  and  Implementing  an  Acquisition  Strategy-  This  process  supports  the 
acquisition  of  products  and  services  by  contract,  such  as  those  organizations  pur¬ 
chasing  or  commissioning  products,  and  especially  important  for  government  agen¬ 
cies,  such  as  the  Department.  Acquisition  of  product  lines  are  typically  structured 
differently  with  the  acquisition  of  the  core  assets  versus  a  product  as  a  contract  de¬ 
liverable,  although  the  role  of  the  architecture  in  a  product  line  may  provide  oppor¬ 
tunities  for  contracting  flexibility  [CN02,  Nor03].  Product  lines  present  several 
challenges  to  the  Department  [Jon99],  requiring  the  development  of  Department 
product  line  acquisition  processes  [BFJ99,  BFG+00,  Cam02,  BOS02],  and  the  shar¬ 
ing  of  lessons-learned  from  successful  Department  product  line  initiatives  [CDS02]. 

•  Funding  -  A  product  line  requires  significant  up-front  investment  to  build  or  ac¬ 
quire  the  core  asset  base,  complete  initial  analysis  efforts,  and  establish  the  produc¬ 
tion  infrastructure;  then  evolving  and  sustaining  the  effort  [CN02,  Nor03],  This  in¬ 
dicates  a  need  for  a  strategic  plan  and  stable,  sustained  funding,  a  significant  chal¬ 
lenge  for  product  lines  in  the  Department. 

•  Launching  and  Institutionalizing  a  Product  Line  -  Launching  and  institutionaliz¬ 
ing  a  product  line  involves  the  initiation  and  improvement  of  the  product  line  prac¬ 
tices  appropriate  for  a  given  organization,  and  viewed  as  a  practice  area  for  apply¬ 
ing  the  other  practice  areas  [CN02,  Nor03]. 

•  Market  Analysis  -  A  market  analysis  is  an  early  step  for  a  product  line,  establishes 
the  product  line  scope,  providing  analysis  on  the  possible  product  line  commonality 
and  variability  [CN02,  Nor03]. 

•  Operations  -  Operations  define  which  part  of  the  organization  develops  the  core  as¬ 
sets  and  products,  and  directs  planning,  processes,  strategies,  policies,  and  con¬ 
straints  [Nor03],  Management  provides  the  resources,  coordination,  and  supervi¬ 
sion,  critical  to  success,  ensuring  coordination  of  the  operations  and  communica¬ 
tions  activities  of  the  product  line  effort  with  a  concept  of  operations  (CONOPS). 

•  Organization  Planning  -  The  product  line  planning  process,  as  mentioned  earlier, 
is  not  unique,  but  does  require  product  line  adoption  plans,  core  asset  funding  plans, 
and  due  to  its  importance,  organization-level  configuration  management  plans 
[CN02,Nor03], 

•  Organizational  Risk  Management  -  Organizational  risk  management  involves  risk 
management  at  the  strategic  level,  and  requires  a  great  deal  of  coordination  across 
project  boundaries  supported  by  open  communication,  integrated  management, 
teamwork,  a  forward-looking  view,  global  perspective,  and  shared  product  vision 
[CN02,  Nor03]. 
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•  Structuring  the  Organization  -  A  product  line  approach  requires  new  roles  and  re¬ 
sponsibilities  supporting  core  asset  development  and  product  development  from  the 
core  asset  base.  An  early  decision  identifies  where  to  locate  the  developers  and 
maintainers  of  the  product  lines  core  assets  and  the  organizations  that  build  the 
product  lines.  Typically,  the  developers  and  maintainers  may  be  located  separately 
from  the  product  line  builders  or  they  may  be  located  together  [CN02,  Nor03], 

•  Technology  Forecasting  -  Technology  forecasts  for  product  lines  include  two  fo¬ 
cus  areas,  internal  development  for  tools,  processes,  or  methods;  or  customer  solu¬ 
tions  such  as  technology  or  solutions  that  may  possibly  affect  product  line  features 
or  capabilities  [CN02,  Nor03], 

•  Training  -  Training  is  a  core  activity  of  organizations  developing  software  and 
supports  the  initial  product  line  adoption  and  the  long-tenn  product  evolution;  and 
requires  management’s  commitment,  an  effective  training  plan,  and  support  of 
product  line  adoption  process  or  process  improvement  [CN02,  Nor03]. 

D.  EVOLVING  SOFTWARE  ARCHITECTURE  THEORY 
1.  The  Federal  Enterprise  Architecture  (FEA) 

The  Office  of  Management  and  Budget  recently  initiated  a  Federal  Enterprise  Ar¬ 
chitecture  (FEA)  program  [OMB02,  FEA02a,  FEA02b,  SAG02,  Hay03b,  BBT+03]  sup¬ 
porting  the  President’s  e-Gov  guidance,  focuses  on  information  technology  investments, 
and  designed  to  facilitate  cross-agency  analysis  and  improvement.  The  lack  of  architecture 
in  the  Federal  Government  is  a  systemic  management  weakness  repeatedly  cited  since  the 
early  1990s  [BBT+03],  The  Federal  Enterprise  Architecture’s  four-layer  segmented 
structure  [FEA02a]  include  systematically  derived  and  captured  structural  descriptions  in 
useful  documentation  (e.g.,  models,  diagrams,  narrative)  for  a  given  enterprise,  including  a 
single  organization,  or  functional  area,  or  a  mission  area  that  transcends  more  than  one  or¬ 
ganizational  boundary  [BBT+03].  The  existing  agency-specific  architectures  will  serve  as 
the  foundation  for  the  FEA,  with  five  reference  models  or  views  under  development  to  as¬ 
sist  with  aligning  existing  data  with  the  FEA, 

•  The  Business  Reference  Model  describes  Federal  Government  business  operations, 
independent  of  individual  agency  implementation, 

•  The  Performance  Reference  Model  provides  a  common  set  of  general  performance 
outputs  and  measures  for  agencies, 


The  four  layers  of  the  FEA  are  the  Technology,  Application,  Data,  and  Business  Architectures  [FEA02a]. 
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•  The  Data  and  Information  Reference  Model  describes  at  an  aggregate  level,  the  data 
types  supporting  agency  operations,  and  the  relationships  among  the  data, 

•  The  Service  Component  Reference  Model  identifies  and  classifies  information  tech¬ 
nology  service  components,  and  promotes  reuse, 

•  The  Technical  Reference  Model  describes  how  technology  supports  delivery  of  ser¬ 
vice  components  and  includes  relevant  standards  [BBT+03]. 

The  Department,  as  an  agency  of  the  Federal  Government  develops  software-intensive  sys¬ 
tems  with  the  intent  of  achieving  interoperability  with  other  stakeholders,  including  other 
Government  agencies  (OGA),  will  support  the  FEA  program. 

The  FEA  suggests  the  use  of  Unified  Modeling  Language  (UML)  [UML99]  for  de¬ 
fining  and  applying  data  models  and  standards  [FEA02a].  In  addition,  XML,  emerging  as  a 
government  and  industry  standard,  provides  a  recommended  foundation  as  the  default  for¬ 
mat  for  moving  and  sharing  highly  structured  infonnation,  as  well  as  less  highly  structured 
information  between  E-Gov  data  architectures  [KJ01,  LE01,  FEA02a],  Data  interoperabil¬ 
ity  principles  for  the  FEA  Data  Architecture  support  improved  interoperability  by, 

•  Avoiding  non-standard  data  syntaxes, 

•  Seeking  industry  vocabularies  before  developing  custom  schemas, 

•  Avoiding  the  “one  size  fits  all”  schema  concept, 

•  Registering  the  semantics  of  shared  data  elements, 

•  Documenting  service  interfaces  in  a  standard  way  [FEA02a]. 

2.  Emerging  Software  Architecture  Concepts 

The  software  architecture  discipline  is  relatively  new,  and  has  not  been  completely 
defined  and  applied  consistently  to  the  life-cycle  management  of  software-intensive  sys¬ 
tems.  Much  of  the  current  software  architecture  research  stems  from  the  earlier  works  of 
[Par72,  Par76,  Par79,  PW92,  SG93,  and  SC96]  and  their  observations  that  software  con¬ 
sists  of  many  structures,  and  that  a  system  is  a  collection  of  related  parts.  Although  there  is 
not  currently  agreement  on  a  precise  definition  of  a  system’s  architecture-  ",  the  IEEE  Rec¬ 
ommended  Practice  for  Architectural  Description  of  Software-Intensive  Systems  [IEEOOb], 
cites  a  consensus  on  the  use  of  multiple  views213,  reusable  specifications  [BS95,  Gac95, 
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Architecture  is  the  fundamental  organization  of  a  system,  embodied  in  its  components,  their  relationship  to  each  other  and  the  envi¬ 
ronment,  and  the  principles  governing  its  design  and  evolution  [IEEOOb] 

213 

a)  A  view  is  a  representation  of  a  whole  system  from  the  perspective  of  a  related  set  of  concerns  [IEEOOb].  b)  A  view  is  a  representa¬ 
tion  of  a  set  of  system  elements  and  the  relationships  associated  with  them  [CBB+03]. 
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HBL97,  MR02]  for  models  within  views,  and  the  relationship  of  architecture  to  the  system 
context.  The  foundation  of  the  approved  [IEEOOb]  builds  on  the  following  concepts  and 
relationships, 

•  Every  software-intensive  system  has  an  architecture,  however,  an  architecture  is  not 
a  system, 

•  An  architecture  and  an  architectural  description  are  not  the  same, 

•  Architecture  standards,  descriptions,  and  development  processes  can  differ  and  be 
separately  developed, 

•  Architecture  descriptions  are  inherently  multi-viewed, 

•  Effective  architecture  description  standards  support  the  separation  of  an  object’s 
view  from  its  specification  [MEH01]. 

The  architecture  of  a  system214  is  a  critical  component  supporting  the  engineering 
process  and  the  life  cycle  model215  of  the  system  [LB95,  SNH95,  MTW96,  CN96,  Fow97, 
HBL97,  LB98,  CWK99,  BBG+00,  BosOO,  IEEOOb,  HHPOO,  MEH01,  MM01,  FG02, 
LLC+02,  MR02,  SB02b,  Fra03,  CBB+03,  GA03,  SPG03,  Tol03],  As  large-scale,  legacy 
software-intensive  systems  evolved,  the  initial  architectural  emphasis  was  on  the  hardware 
and  network  architecture  components  of  the  information  systems,  until  the  complexity  of 
the  software  technology  and  the  cost  of  software  development  necessitated  a  change  in  the 
emphasis  to  include  the  software-related  architecture  issues  [ShaOl]  of  today’s  software¬ 
intensive  system.  Software  architecture  especially  for  large,  software-intensive  systems, 

•  Serves  as  the  system  blueprint,  a  focal  point  for  the  project  development  team  and 
mutual  communication, 

•  Serves  as  the  foundation  for  the  system’s  quality  attributes, 

•  Provides  the  first  artifact  for  early  design  decisions  indicating  the  system  meets 
requirements, 

•  Supports  a  transferable  abstraction  of  the  system  for  activities  including  post¬ 
deployment  maintenance  or  mining  [CN96,  BBG+00,  BOSOO,  MR02]. 

The  software  architecture  is  key  to  the  success  of  any  software  project  [LB98, 
BCK98]  and  critical  to  the  success  of  a  product  line  initiatives  [Jon99,  BosOO,  DSOO, 

ProOO,  ShaOOa,  MorOOa,  DonOO,  CN02].  Architectural  drivers,  influencing  the  entire  life 


System  is  defined  as  a  collection  of  components  organized  to  accomplish  a  specific  function  or  set  of  functions  [IEEOOb]. 

215 

The  life  cycle  model  is  a  framework  containing  the  processes,  activities,  and  tasks  involved  in  the  development,  operation,  and  main¬ 
tenance  of  a  software  product,  which  spans  the  life  of  the  system  from  the  definition  of  its  requirements  to  the  termination  of  its  use 
[IEEOOb]. 
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cycle  of  a  system,  include  quality  attributes,  business  goals,  and  system  interactions 
[MR02],  Product  line  architectures  also  include  the  allowable  product  variability  and  the 
different  instantiations  of  the  product,  since  these  too  are  products. 

There  may  also  be  many  views  in  a  software  architecture  domain,  which  show  spe¬ 
cific  properties  of  the  software  system  [IEEOOb].  These  different  architecture  views  ad¬ 
dress  diverse  issues  encountered  with  the  design  process,  including  the  logical  view,  proc¬ 
ess  view,  physical  view,  and  the  development  view.  Other  terms  of  references  for  software 
architecture  design  views  include  behavioral,  functional,  structural,  and  data  modeling 
views,  however,  the  key  point  made  by  [BBOO]  is  that  software  architecture  design  is  the 
product  of  a  multi-dimensioned  process  composed  of  independent  and  orthogonal  views, 
impacted  by  variability,  either  planned  or  unintended. 

[CBB+03]  defines  an  architectural  style"  (e.g.,  pattern)  as  a  specialization  of  ele¬ 
ment  and  relation  types,  together  with  a  set  of  constraints  on  their  use.  In  this  research  ef¬ 
fort,  an  architectural  style  consists  of  a  set  of  constraints  on  the  architecture,  which  define 
the  set  or  family  of  architectures,  and  satisfy  them  with  a  number  of  major  styles.  These 
styles  may  include  general  structures  of  pipes  and  filters  [SC96],  layers  [BBG+00],  black- 
boards,  object-orientation,  and  implicit  invocation"  [BosOO];  distributed  systems;  interac¬ 
tive  systems;  adaptable  systems;  and  other  styles  including  batch,  interpreters,  process  con¬ 
trol,  specification-based,  and  rule-based. 

Software  architecture  theory  is  a  rapidly  evolving  component  of  software  engineer¬ 
ing  with  a  wide  range  of  diverse  concepts,  styles,  and  views  including:  the  development  of 
domain  specific  repository  for  components  [Gac95,  HBL97],  structural  modeling  [CB96], 
software  architecture  overview  [CN96],  transitioning  to  a  model-based  engineering  archi¬ 
tectural  style  [GP96],  and  includes  industrial  best-practices  for  evaluating  software  archi¬ 
tectures  [ABC+97].  The  evolving  software  architecture  theory  body  of  knowledge  includes 
a  wide  spectrum  of  topics:  evaluating  the  quality  attributes  of  a  software  architecture 
[BKW97],  reconstructing  a  software  architecture  with  automated  support  [KC97],  reusable 


Shaw  and  Clements  define  architectural  styles  as  a  set  of  design  rules  that  identify  the  kinds  of  components  and  connectors  that  may 

be  used  to  compose  a  system  or  subsystem,  together  with  local  or  global  constraints  on  the  way  the  composition  is  done  [SC96]. 
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Mary  Shaw  and  David  Garlan  introduced  implicit  invocation  in  (1996)  as  a  style  that  organizes  the  system  in  terms  of  components 
that  generate  events,  possibly  containing  data,  and  that  consume  events.  An  example  is  the  JavaBeans  standard  [BosOO]. 
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components  implemented  via  a  relational  hydrograph  model  [Luq90,  HBL97,  LG97],  soft¬ 
ware  architectural  transformation  via  automated  code  transformation  [CWK99],  interop¬ 
erability  [Sut99],  attribute -based  architectural  styles  [KK99],  and  methods  for  documenting 
architectural  layers  [BBG+00]. 

3.  Software  Architecture  Views 

218* 

A  system"  is  a  collection  of  components  organized  to  accomplish  a  specific  func¬ 
tion  or  set  of  functions  [IEEOOb].  In  addition  to  defining  architecture,  [PW92]  emphasized 
certain  architectural  considerations  for  different  stakeholders  or  for  different  uses  with  a 
variety  of  system  views.  A  sample  of  other  recent  advances  in  software  architectures  noted 
in  Table  8-2  include  [Kru95]  describing  four  views  of  software  architecture  for  system 
building;  the  collaborative  work  of  [SNH95]  who  identified  four  additional  industrial  use 
views;  four  business  views  [HSOOc],  the  development  of  viewpoints219  [IEEOOb]  to  desig¬ 
nate  the  means  used  to  construct  individual  views;  and  the  three  categories  of  views  identi¬ 
fied  as  viewtypes220  [CBB+03]. 
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Behavioral 
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Code 

Functional 

Link  Bit 

Scenarios  +1 

Table  8-2.  Software  Architectural  Views  /  Viewpoints  /  Viewtypes  (From  [CBB+03]) 


2 1 8  A  system  also  includes  individual  applications,  traditional  systems,  subsystems,  systems  of  systems,  product  lines,  product  families, 
whole  enterprises,  and  other  aggregated  types  [IEEOOb]. 
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A  viewpoint  is  specification  of  the  conventions  for  constructing  and  using  a  view.  A  pattern  or  template  from  which  to  develop 

individual  views  by  establishing  the  purposes  and  audience  for  a  view  and  the  techniques  for  its  creation  and  analysis  [IEEOOb]. 

220 

A  viewtype  defines  the  element  types  and  relationship  types  used  to  describe  the  architecture  of  a  software  system  from  a  particular 
perspective  [CBB+03]. 
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There  have  been  many  attempts  to  overcome  the  formidable  risks  and  difficulties 
experienced  in  the  design,  development,  deployment,  and  evolution  of  software-intensive 
systems,  with  improved  software  engineering  practices,  procedures,  and  techniques 
[MR02],  In  1996,  the  IEEE  chartered  the  Architecture  Working  Group  (AWG)  to  imple¬ 
ment  the  approved  recommendations  from  the  1995  IEEE  Architecture  Planning  Group  for 
software-intensive  systems.  The  [IEEOOb]  charter  tasked  the  AWG  to  define  tenns,  princi¬ 
ples  and  guidelines  for  the  consistent  application  of  architectural  precepts  for  systems 
throughout  the  entire  life  cycle. 

The  AWG  elaborated  on  architectural  precepts  and  potential  benefits  for  software 
products,  systems,  and  aggregated  systems  (e.g.,  systems  of  systems);  provided  a  frame¬ 
work  for  architectural  attributes;  and  developed  a  roadmap  for  architectural  precepts  in  the 
generation,  revision  and  application  of  IEEE  standards.  In  developing  [IEEOOb],  the  AWG 
intended  to  capture  the  architectural  information  contained  in  the  various  products  of  the 
system  development  process  illustrated  in  Figure  8-1  and  devise  an  architectural  descrip¬ 
tion221: 

•  Expressing  the  system  and  its  evolution, 

•  Supporting  communication  among  the  system  stakeholders, 

•  Allowing  a  consistent  comparison  of  architectures, 

•  Supporting  system  development, 

•  Identifying  the  system’s  persistent  characteristics  and  supporting  principles  for  fu¬ 
ture  changes, 

•  Verifying  the  system’s  implementation  as  compliant  with  an  architectural  descrip¬ 
tion, 

•  Recording  contributions  to  the  body  of  knowledge  of  software-intensive  system  ar¬ 
chitectures  [IEEOOb]. 


An  architectural  description  is  a  collection  of  products  to  document  an  architecture  [IEEOOb]. 
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Figure  8-1.  Model  of  an  Architectural  Description  (From  [IEEOOb]) 

4.  Software  Architecture  Quality  Analysis  Methods 

[DN02]  suggest  that  over  the  past  several  years,  software  architecture  emerged  as 
the  appropriate  level  for  addressing  software  quality,  and  recent  efforts  to  understand  the 
design  patterns  and  architectural  styles  contribute  to  that  quality  analysis  effort.  In  their 
research  [DN02]  provide  a  concise  and  recommended  survey  on  eight  representative  soft¬ 
ware  analysis  methods  in  different  domains,  compare  and  contrast  similarities  and  differ¬ 
ences  through  study,  comparison,  and  classification,  and  offer  guidelines  supporting  the  use 
of  the  most  suitable  method  for  an  architecture  assessment  process. 

[DN02]  fit  these  eight  methods  into  three  categories  or  communities:  software  met¬ 
rics,  scenario-based,  and  attribute  model-based  analysis.  The  software  metrics  analysis 
technique  uses  module  coupling  and  cohesion  theories  or  more  abstract  evaluations  to  de¬ 
fine  predictive  measures,  of  quality;  while  scenario-based  methods  applied  over  the  past 
several  years  have  a  maturity  and  a  certain  level  of  face  validation  based  on  use;  while  the 

attribute  model-based  is  too  new  to  evaluate  [DN02].  These  eight  methods  include: 
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•  Scenario-Based  Architecture  Analysis  Method  (SAAM)  supports  a  general  under¬ 
standing  of  the  general  architectural  concepts  supporting  proof  that  the  software 
system  meets  more  than  the  functional  requirements, 

•  SAAM  Founded  on  Complex  Scenarios  (SAAMCS)  considers  that  the  complexity 
of  scenarios  is  the  most  important  factor  for  risk  assessment, 

•  Extending  SAAM  by  Integration  in  the  Domain  (ESAAMI)  a  combination  of  ana¬ 
lytical  and  reuse  concepts,  integrates  SAAM  in  the  domain-specific  and  reuse-based 
development  process,  considering  only  the  problem  description,  requirements 
statement,  and  architecture  description, 

•  Software  Architecture  Analysis  Method  for  Evolution  and  Reuse  (SAAMER),  an 
extension  of  SAAM  assessed  quality  objectives  based  on  two  attributes,  evolution 
and  reusability, 

•  The  Architecture  Trade-off  Method  (AT AM)  an  attribute-based  model,  provides  a 
way  of  understanding  a  software  architecture’s  capability  with  multiple,  competing 
quality  attributes, 

•  Scenario-Based  Architecture  Reengineering  (SBAR)  evaluates  multiple  quality  at¬ 
tributes  of  the  architecture  design  in  a  scenario-based  review  of  the  software  quali¬ 
ties  of  a  specific  software  architecture  or  system, 

•  Architecture  Level  Prediction  of  Software  Maintenance  (ALPSM)  analyzes  main¬ 
tainability  of  a  software  system  employing  scenarios  at  the  software  architecture 
level  and  using  the  size  of  the  change  as  a  predictor, 

•  Software  Architecture  Evaluation  Model  (SAEM),  a  quality  attribute  model,  estab¬ 
lishes  the  basis  for  the  software  architecture  internal  and  external  quality  evaluation 
and  prediction  of  final  system  quality  [DN02]. 

Software  quality  is  one  of  three  dependent,  user-oriented  product  characteristics,  in 
addition  to  cost  and  schedule  [BKW97].  A  process  mature  organization  may  predict  and 
control  cost,  however,  process  maturity  does  not  automatically  translate  into  product  qual¬ 
ity,  which  requires  mature  technology  to  predict  and  control  quality  attributes  [BKW97]. 
[BKW97,  BKBOO,  BKB01,  and  BBK02]  describe  the  quality  attribute  requirements  for 
performance,  security,  modifiability,  reliability,  and  usability  and  the  significant  influence 
of  these  attributes  for  evaluating  software  architectures. 

Other  representative  software  architecture  quality  assessment  techniques  include  the 
Model-Driven  Architecture  Theory  [SieOl,  Fra03],  supported  by  the  Object  Management 
Group,  simulation,  mathematical  modeling,  and  experienced-based  assessment  techniques 
[BosOO],  [CBB+03]  identified  Department  software  architecture  theory  as  a  fledgling  prac¬ 
tice  and  considers  the  JTA  and  C4ISR  architecture  framework  focus  on  the  system  archi¬ 
tecture,  and  HLA  publish-subscribe  mechanism  lacking  in  software  architecture  substance. 
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However,  an  understanding  of  software  architecture  in  the  M&S  domain  continues  to  grow, 
and  [Tol02,  Tol03]  discusses  future  architecture  requirements  for  HLA  development.  The 
SEI  also  contributed  the  Attribute-Based  Architecture  Style  (ABAS)  [KK99],  Architecture 
Based  Design  (ABD)  [BK99]  and  Attribute  Driven  Design  (ADD)  [BKBOO]  methods. 

The  Architecture  Based  Design  (ABD)  method  [BK99,  BBC+00,  BBKOO]  provides 
a  recursive  framework  with  three  foundations  of  attributes  (e.g.,  functional,  quality,  and 
business  requirements)  at  a  level  of  abstraction  supporting  the  necessary  variation  for  pro¬ 
ducing  products,  and  includes  function  decomposition,  architectural  styles,  and  software 
templates  for  designing  high-level  software  architectures.  [BKB01]  develops  functional, 
quality  and  business  requirements,  or  architectural  drivers,  at  a  level  of  abstraction  that  al¬ 
lows  for  the  variation  needed  to  produce  specific  products  for  a  product  line  or  any  type  of 
system  with  a  long  lifeline.  The  ABD  method  [BK99,  BBC+00,  BBKOO]  provides  a  proc¬ 
ess  for  designing  the  conceptual  software  architecture,  support  for  organization  functions, 
identification  of  synchronization  points,  and  allocation  of  functions  to  processes,  conclud¬ 
ing  with  allocation  commitments  to  classes,  processes  or  operating  systems.  A  product  of 
the  ABD  process  [BBC+00]  is  a  collection  of  software  templates  ,  which  constrain  the 
implementation  of  different  types  of  components  with  a  description  of  component  interac¬ 
tions  and  responsibilities. 

Another  SEI-sponsored  approach  for  defining  a  software  architecture  based  on  a  de¬ 
sign  process  driven  with  specified  functional  and  quality  attribute  requirements  possessed 
by  the  software  is  the  Attribute  Driven  Design  (ADD)  [BKBOO,  BKB01,  BBK02]  method, 
a  recursive,  decomposition  process  where  at  each  stage  of  the  decomposition  process,  cho- 
sen  attribute  primitives  ~  [BKBOO]  attempt  to  satisfy  a  set  of  quality  scenarios  [BKBOl], 

In  addition  the  development  of  quality  architectures  necessitates  early  consideration  of  fac¬ 
tors  affecting  the  various  quality  attributes  and  the  impact  on  software  design  [BKBOO, 
BBK02],  [BKBOO,  BKBOl,  and  BBK02]  further  suggest  the  use  of  general  scenarios  to 
support  development  of  quality  attributes,  with  each  general  scenario  consisting  of: 
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A  software  template  defines  the  software  element  of  a  particular  type,  including  patterns  describing  interactions  with  shared  services 

and  infrastructure,  and  citizenship  responsibilities  [BBC+00]. 
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An  attribute  primitive  is  a  collection  of  components  and  connectors  that  1)  collaborate  to  achieve  some  quality  attribute  goal,  nor¬ 
mally  expressed  in  a  general  scenario;  and  2)  is  minimal  with  respect  to  the  achievement  of  these  goals  [BKBOl]. 
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•  The  stimuli  requiring  an  architecture’s  response, 

•  The  source  of  the  stimuli, 

•  The  context  in  which  the  stimuli  occurs, 

•  The  type  of  system  elements  involved  in  the  response, 

•  Possible  responses, 

•  The  measure’s  employed  to  characterize  the  architecture’s  response  [BKBOO, 

BKB01,  BBK02]. 

An  example  of  a  reliability  general  scenario  is  the  case  where  an  internal  component  fails. 
The  system  recognizes  a  failure  of  an  internal  component  and  possesses  capabilities  to 
compensate  for  the  fault.  The  SEI  developed  quality  attribute  workshops  as  a  method  to 
analyze  a  system  against  a  number  of  critical  quality  attributes  [BW02,  BEL+02]. 

Quality  attribute  design  primitives  are  basically  templates  and  provide  building 
blocks  for  developing  architecture  designs  supporting  the  achievement  of  specific  quality 
attribute  goals  [BKB01],  An  attribute  primitives  addresses  one  or  more  quality  attributes 
characterized  by  one  or  more  general  scenarios  [BKB01].  In  a  product  line  different  prod¬ 
ucts  may  have  different  quality  attribute  requirements  or  the  products  may  exist  simultane¬ 
ously  and  only  vary  in  terms  of  different  attributes,  characteristics,  or  scale  factors. 

5.  The  extensible  Markup  Language  (XML) 

The  World  Wide  Web  Consortium  (W3C)  developed  the  key  technical  standards  for 
XML,  including  Namespaces  the  XML  Information  Set,  and  XML  Inclusions  [Sal02]. 
XML,  as  an  application  profile  or  restricted  fonn  of  the  Standard  Generalized  Markup 
Language  (SGML)  describes  a  class  of  data  objects  called  XML  documents  and  partially 
describes  the  behavior  of  programs  which  process  them  [Sal02].  Storage  units  called  enti¬ 
ties,  which  contain  parsed  or  unparsed  data,  comprise  XML  documents.  Parsed  data  in¬ 
cludes  characters,  which  form  either  character  data  or  markup.  Markup  encodes  a  descrip¬ 
tion  of  the  document’s  storage  layout  and  logical  structure  [Sal02]. 

Expanding  areas  of  XML  research  include  W3C  working  group  efforts  for  an  XML 
Query  (XQuery)  data  model  [CFM+03],  and  query  language  [FMM+03].  XML  application 
areas  include:  database  interoperability  [HinOl],  heterogeneous  software  systems  [YouOl, 
TA01],  XML  schema  integration  [HalOlb],  real-time  system  data  interchange  [PraOl], 
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common  data  attributes  [ZobOl],  an  XML  technology  assessment  [BerOl],  and  workflow 
and  document  management  [ACD+02], 

In  the  M&S  area,  XML  uses  include:  data  interchanges  for  equipment  and  perform¬ 
ance  information  [LKB01,  LPA+02],  support  of  command  and  control  communications 
requirements  [SBOO],  data  management  and  architecture  description  [SB02b],  the  devel¬ 
opment  of  integration  and  collaborative  toolsets  [GRG+02],  support  to  major  simulation 
development  efforts  [And02],  and  scenario  generation  [RK02],  Web-based  M&S  tech¬ 
niques  employing  XML  for  development  and  interoperability  have  also  emerged  [BZP+02, 
Hob03,  Tol03], 

However,  [MCF+02]  identify  challenges  in  realizing  XML’s  full  potential  including 
risks  that  data  will  lack  definition,  incompatible  data  definitions,  and  the  proliferation  of 
vocabularies  and  structures.  [BerOl]  suggest  that  the  lack  of  agreement  on  data  representa¬ 
tions  and  conceptual  data  models  represent  a  major  obstacle  to  data  interchange  among  leg¬ 
acy  systems,  but  concludes  that: 

Data  interoperability  is  feasible  without  requiring  a  comprehensive  data 
standard,  and  that  methods  for  incrementally  growing  localized  standards 
and  bridging  the  gaps  among  them  without  requiring  global  agreement  ap¬ 
pear  possible.  Further  assessments  are  proposed  to  determine  the  practical 
feasibility  of  applying  this  approach  to  DoD  operations  [BerOl]. 

6.  Software  Architecture  Description  Languages  (ADL) 

Various  Architectural  Description  Languages  (ADL)  support  current  software  ar¬ 
chitecture  research  and  development  efforts  with  the  potential  for  employing  common 
coarser-grained  architectural  elements  (e.g.,  components  and  connectors)  and  interconnec¬ 
tivity  schemes  for  architecture-based  development,  and  featuring  formal  modeling  nota¬ 
tions,  analysis,  and  development  tools  operating  on  architectural  specifications  [MTOO]. 
However,  the  research  community  currently  lacks  consensus  on  ADLs,  the  capabilities  ex¬ 
pected  from  an  ADL  toolset,  and  what  aspects  of  a  software  architecture  to  model.  [MTOO, 
CBB+03]  suggest  that  no  existing  ADL  tool  provides  the  complete  capability  to  document 
software  architectures,  although  many  ADLs  perform  well  in  specific  areas  such  as  concep¬ 
tual  frameworks,  concrete  syntax,  parsing,  displaying,  compiling,  analyzing,  or  simulating 


212 


architectural  descriptions  in  specific  languages.  While  there  is  no  generally  accepted  defi- 
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nition~~  of  an  ADL,  [MTOO]  proposes  a  definition  and  classification  for  ADLs: 

An  ADL  must  explicitly  model  components,  connectors,  and  their  configu¬ 
rations;  furthennore,  to  be  truly  usable  and  useful,  it  must  provide  tool  sup¬ 
port  for  architecture-based  development  and  evolution.  These  four  elements 
of  an  ADL  are  further  broken  down  into  constituent  parts  [MTOO]. 

Although  lacking  a  consistent  definition,  [MTOO,  CBB+00]  cite  several  do¬ 
main-specific  and  general  purpose  ADLs  including  ACME,  Aesop,  Darwin,  MetaH, 
Rapide,  SADL,  UniCon,  and  Wright.  Emphasizing  the  lack  of  a  standard  definition 
and  scope,  [MTOO]  describe  ACME  as  an  architectural  interchange  language  ena¬ 
bling  integration  of  support  tools  across  ADLs,  while  [CBB+00]  categorize  ACME 
as  an  ADL.  [MTOO]  provide  a  succinct  ADL  classification  and  comparison  frame¬ 
work  of  key  ADL  properties,  identifying  capabilities  and  deficiencies. 

E.  SUMMARY 

Chapter  VIII  discussed  traditional  product  lines,  and  introduces  software  product 
lines  and  software  architectures.  The  chapter  also  reviews  core  asset  development,  reverse 
engineering  to  develop  core  assets,  reengineering  core  assets,  component  development  sup¬ 
porting  a  product  line  methodology,  product  line  practice  areas,  and  an  overview  of 
evolving  software  architecture  theory  potentially  applicable  to  improved  M&S  credibility 
and  reducing  heterogeneous  system  representation  anomalies. 

Product  lines  and  product  line  practices  are  new  to  software  engineering,  the  De¬ 
partment,  and  the  Department  M&S  domain.  Product  line  practice  areas  include  software 
engineering,  technical  management  and  organizational  management,  which  may  be  appli¬ 
cable  to  meet  some  or  all  of  the  technical,  cultural,  and  managerial  challenges  collectively 
hindering  the  development  of  improved  Department  M&S  credibility. 

A  product  line  approach  to  developing  and  deploying  software-intensive  systems 
offers  great  promise  for  delivering  higher  quality  systems  in  a  shorter  time  and  at  reduced 
cost.  A  product  line  approach  to  software-intensive  systems  can  save  money  and  result  in  a 
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Another  definition  of  an  ADL  is  a  language  (graphical,  textual,  or  both)  for  describing  a  software  system  in  terms  of  its  architectural 
element  and  the  relationships  among  them  [CBB+00]. 
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faster  time  to  field  quality  systems.  A  number  of  organizations  gained  order-of-magnitude 
improvements  in  efficiency,  productivity,  and  quality  through  a  product  line  approach. 
Product  line  practice  also  enables  an  organization  to  get  its  product  to  the  market  more  rap¬ 
idly.  As  an  example,  [BC96]  provides  a  case  study  of  a  successful  product  line  implemen¬ 
tation  by  the  CelsiusTech  Systems  AB  of  Sweden  in  the  area  of  large,  embedded  real-time 
shipboard  command  and  control  systems. 

Any  software  product  line  implementation  strategy  in  the  Department  must  take 
into  account  the  Department  is  an  acquirer  of  software  systems  and  not  the  software  devel¬ 
oper,  and  factor  in  product  line  practice  areas  supporting  the  Department’s  acquisition 
process.  Initial  efforts  to  employ  product  lines  in  the  Department’s  M&S  domain  have 
been  limited. 

Components  play  a  major  role  in  many  software-intensive  systems.  There  are  sev¬ 
eral  motivations  for  a  components-based  software  approach  to  software:  providing  a  basis 
for  commerce  in  reusable  software  to  meet  demand,  facilitating  the  development  of  flexible 
systems,  and  reducing  the  time  to  design,  implement,  and  deploy  systems.  As  an  indication 
of  the  need  for  software  components  at  the  United  States  Federal  Government  level,  the 
Office  of  Management  and  Budget  recently  established  the  Federal  Enterprise  Architecture 
Program  Management  Office  to  remedy  the  lack  of  a  Federal  Enterprise  Architecture 
(FEA),  a  major  barrier  to  the  success  of  the  E-Govemment  initiative.  Currently  there  is  not 
an  established  basis  for  how  well  component  models  and  frameworks  contribute  to  achiev¬ 
ing  the  desired  quality;  nor  is  there  any  basis  for  assessing  the  quality  of  software  compo¬ 
nent  interfaces,  composition,  and  component  certification.  Consensus  on  the  many  solu¬ 
tions  required  for  contentious  component  /  composition  issues  do  not  appear  on  the  near- 
term  horizon. 

A  software  product  line  practice  area  is  a  body  of  work  or  a  collection  of  activities, 
which  an  organization  must  successfully  master  to  carry  out  the  essential  work  of  a  soft¬ 
ware  product  line.  Many,  if  not  the  majority  of  the  practice  areas  describe  activities,  which 
are  critical  to  the  success  of  any  software  project,  and  provide  starting  points  for  organiza¬ 
tions  to  initiate  and  master  activities  and  measure  progress  in  adopting  a  product  line  ap¬ 
proach.  The  practice  areas  in  the  framework  divide  into  three  categories:  1)  software  engi- 
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neering  practices,  2)  technical  management  practices,  and  3)  organizational  management 
practices  illustrated  in  Table  8-1. 

Implementing  software  product  lines  offers  potential  gains,  and  entails  significant 
risks,  including  technical,  organizational,  and  management  issues.  [Nor03]  explains  that 
building  and  acquiring  a  software  product  line  requires  disciplined  engineering  supported 
by  mature  technical  and  organization  management  processes  of  universal  essential  activi- 
ties  and  practices,  which  the  SEI  developed  into  an  online  framework  ,  a  web-based 
document  describing  the  competencies  needed  to  develop  and  field  a  successful  a  software 
product  line  or  any  software-intensive  system.  Based  on  research  to  date,  the  SEI  proposes 
the  software  engineering  practices  listed  in  Table  8-1  as  necessary  practices  supporting  an 
organization’s  capability  and  technology  to  create,  evolve,  and  maintain  core  assets  and 
products. 

Software  architecture  theory  is  a  rapidly  evolving  component  of  software  engineer¬ 
ing  with  a  wide  range  of  diverse  concepts,  styles,  and  views.  The  evolving  software  archi¬ 
tecture  theory  body  of  knowledge  includes  a  wide  spectrum  of  topics:  evaluating  the  qual¬ 
ity  attributes  of  a  software  architecture,  reconstructing  a  software  architecture  with  auto¬ 
mated  support,  reusable  components,  software  architectural  transfonnation,  interoperabil¬ 
ity,  attribute-based  architectural  styles,  new  XML  architecture-focused  techniques  and  ap¬ 
plications,  emerging  architecture  description  languages,  and  methods  for  documenting  ar¬ 
chitectural  layers. 

The  software  architecture  is  key  to  the  success  of  any  software  project  and  critical 
to  the  success  of  a  product  line  initiatives.  Architectural  drivers,  influencing  the  entire  life 
cycle  of  a  system,  include  quality  attributes,  business  goals,  and  system  interactions.  Prod¬ 
uct  line  architectures  also  include  the  allowable  product  variability  and  the  different  instan¬ 
tiations  of  the  product,  since  these  too  are  products.  There  may  also  be  many  views  in  a 
software  architecture  domain,  which  show  specific  properties  of  the  software  system. 

These  different  architecture  views  address  diverse  issues  encountered  with  the  design  proc¬ 
ess,  including  the  logical  view,  process  view,  physical  view,  and  the  development  view. 
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Framework  in  this  context  suggests  a  conceptual  index,  a  frame  of  reference  for  the  information  essential  for  success  with  product 
lines  [Nor03]. 
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Other  terms  of  references  for  software  architecture  design  views  include  behavioral,  func¬ 
tional,  structural,  and  data  modeling  views,  however,  the  key  point  is  that  software 
architecture  design  is  the  product  of  a  multi-dimensioned  process  composed  of  independent 
and  orthogonal  views,  impacted  by  variability,  either  planned  or  unintended. 

An  architectural  style  (e.g.,  pattern)  is  a  specialization  of  element  and  relation  types, 
together  with  a  set  of  constraints  on  their  use.  In  this  research  effort,  an  architectural  style 
consists  of  a  set  of  constraints  on  the  architecture,  which  define  the  set  or  family  of  archi¬ 
tectures,  and  satisfy  them  with  a  number  of  major  styles.  These  styles  may  include  general 
structures  of  pipes  and  filters,  layers,  blackboards,  object-orientation,  and  implicit  invoca¬ 
tion,  distributed  systems;  interactive  systems;  adaptable  systems;  and  other  styles  including 
batch,  interpreters,  process  control,  specification-based,  and  rule-based. 

A  sample  of  other  recent  advances  in  software  architectures  noted  in  Table  8-2  in¬ 
clude  [Kru95]  describing  four  views  of  software  architecture  for  system  building;  the  col¬ 
laborative  work  of  [SNH95]  who  identified  four  additional  industrial  use  views;  four  busi¬ 
ness  views  [HSOOc],  the  development  of  viewpoints  [IEEOOb]  to  designate  the  means  used 
to  construct  individual  views;  and  the  three  categories  of  views  identified  as  viewtypes 
[CBB+03].  [DN02]  suggest  that  over  the  past  several  years,  software  architecture  emerged 
as  the  appropriate  level  for  addressing  software  quality,  and  that  recent  systemic  efforts  of 
understanding  how  the  design  patterns  and  architectural  styles  contribute  to  that  quality 
analysis  effort.  In  their  research  [DN02]  provide  a  concise  and  recommended  survey  on 
eight  representative  software  analysis  methods  in  different  domains,  compare  and  contrast 
similarities  and  differences  through  study,  comparison,  and  classification,  and  offer  guide¬ 
lines  supporting  the  use  of  the  most  suitable  method  for  an  architecture  assessment  process. 

Emerging  software  tools  including  XML  and  ADLs  support  current  software  archi¬ 
tecture  research  and  development  efforts  with  the  potential  for  employing  common  coarser- 
grained  architectural  elements  (e.g.,  components  and  connectors)  and  interconnectivity 
schemes  for  architecture -based  development,  and  featuring  formal  modeling  notations, 
analysis,  and  development  tools  operating  on  architectural  specifications.  However,  the 
research  community  currently  lacks  consensus  on  ADLs,  the  capabilities  expected  from  an 
ADL  toolset,  and  what  aspects  of  a  software  architecture  to  model. 
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IX.  ANALYSIS  OF  MODEL  CREDIBILITY  IN  FIVE  BMDS  M&S 


A.  INTRODUCTION 

Chapter  IX  provides  an  overview  of  the  Ballistic  Missile  Defense  System  (BMDS), 
domain  M&S,  a  concise  description  of  the  BMDS  domain  background226,  the  BMDS  do¬ 
main  M&S  hierarchy,  a  BMDS  System-Level  M&S  synopsis,  BMDS  M&S  demographics, 
and  a  review  of  the  BMDS  model  representations  populating  the  five  BMD  System-Level 
simulations  under  study  in  this  dissertation.  The  BMD  domain  overview  section  provides 
the  Agency  background  and  highlights  the  significant  role  Department  organizational 
changes  played  in  the  current  status  of  BMD  System-Level  M&S.  Building  on  the  organ¬ 
izational  outline  and  M&S  domain  overview,  the  chapter  continues  with  a  review  on  the 
Agency’s  implementation  of  the  Department’s  policies  for  establishing  simulation  credibil¬ 
ity  supporting  confidence  in  BMDS  simulation  results,  and  VV&A  background  infonnation 
for  each  of  the  BMD  System-Level  M&S. 

Summary  level  information  of  the  other  M&S  in  the  domain  M&S  hierarchy  pro¬ 
vide  additional  context  for  the  analysis.  A  top-level  review  of  the  BMD  System-Level 
M&S  fidelity,  and  the  foundations  for  radar  sensor  [Mac92,  Cla93,  EB01]  fidelity  follow. 
The  analysis  identifies  additional  root  causes  for  heterogeneous  anomalies  in  the  BMD 
System-Level  M&S.  The  research  methodology  supported  by  the  NPS  software  engineer¬ 
ing  distance-learning  model  facilitates  the  timely  study  of  Department  primary  source  ma¬ 
terial  for  software-intensive  legacy  simulation  systems.  This  case  study  also  employs  se¬ 
lected  product  line  practice  areas  (e.g.,  Organization  Management,  Technical  Management, 
Software  Engineering)  [Coh02]  as  a  tailored  framework  for  the  missile  defense  domain 
analysis.  The  scope  of  this  research  includes  five  large-scale,  Missile  Defense  Agency  leg¬ 
acy  simulations,  identified  later  in  the  chapter,  and  supporting  appendices. 


226  The  JTA  includes  specific  functional  domains,  separate  from  the  JTA  Core  standards  and  guidelines  and  which  may  be  inappropriate 
for  systems  in  other  domains,  only  to  ensure  interoperability  within  the  domain.  These  domains  include  subdomains  containing  JTA 
elements  applicable  to  systems  within  that  subdomain.  The  intention  of  the  JTA  is  for  the  systems  in  the  subdomains  and  domains  to 
eventually  adopt  the  JTA  elements  in  the  JTA  Core  standards  [JTA02a]. 
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B. 


SELECTION  CRITERIA  OF  A  RELEVANT  DOD  DOMAIN 


997 

The  selected  Department  M&S  domain"  ,  the  Ballistic  Missile  Defense  System 
(BMDS),  met  the  following  research  criteria:  1)  the  BMDS  domain  is  instantiated  in  the 
Department’s  Joint  Technical  Architecture  [JTA02a]  specifically  in  the  weapon  system 
(e.g.,  missile  defense  sub-domain),  M&S,  and  C4ISR“  (e.g.,  space  reconnaissance  sub- 
domain)  domains  identified  by  the  bold  areas  in  Figure  9-1;  3)  the  selected  BMDS  domain 
is  also  currently  engaged  with  multiple  international  cooperative  M&S  programs;  4)  the 
BMDS  domain  is  part  of  the  Department’s  Transformation  process  [RumOl]  with  trans¬ 
formation  criteria  established  by  [Rum02c,  Ald02e];  5)  the  BMDS  is  a  high-risk,  mission- 
critical  defense  capability  (Figure  9-2)  for  the  Nation,  allies,  and  friends;  2)  the  Missile  De¬ 
fense  Agency  (MDA),  responsible  for  the  BMDS  is  a  jointly  organized  Department  domain 
including  Army,  Navy,  and  Air  Force  systems;6)  the  BMDS  domain  will  implement  the 
Department’s  evolutionary  acquisition  strategy  [Ald02d];  with  the  acquisition  strategy  cri¬ 
teria  established  by  [Rum02c,  Ald02e];  and  7)  the  BMDS  domain  possessed  an  established 
hierarchy  of  M&S,  introduced  in  Figure  9-3,  capable  of  supporting  an  analysis  of  the  De¬ 
partment’s  legacy  architecture  for  reverse  engineering,  reengineering,  and  reuse  (R3). 


Figure  9- 1 .  JTA  Domains  Included  in  this  Study  (After  [JTA02a]) 
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In  this  context  a  domain  is  an  area  of  knowledge  or  activity  characterized  by  a  set  of  concepts  and  terminology  understood  by  practi¬ 
tioners  in  that  area. 

“8  The  acronym  C2/BM  represents  the  BMDS  C4ISR  function  in  this  work. 
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C.  ORGANIZATION  MANAGEMENT 

1.  Domain  Overview  of  the  Organization  Structure 

In  order  to  meet  the  changing  ballistic  missile  threat  to  the  Nation  and  adapt  from  a 
Cold  War  monolithic  threat  environment  to  the  current  Post-Cold  War  asymmetrical 
threats,  the  Nation  has  continued  to  evolve  its  missile  defense  capabilities.  This  evolution 
began  with  President  Reagan’s  Strategic  Defense  Initiative  in  1983,  which  established  the 
Strategic  Defense  Initiative  Organization  (SDIO),  continued  with  the  establishment  of  its 
successor,  the  Ballistic  Missile  Defense  Organization  (BMDO)  [BMDOOa],  and  most  re¬ 
cently,  the  current  Missile  Defense  Agency  (MDA)  [Rum02c,  Ald02e].  As  the  world  geo¬ 
political  situation  changed  in  the  1980s  and  1990s,  national-level  policy  and  defense  strat¬ 
egy  reordered  the  priorities  and  focus  of  the  Agency  several  times. 

In  addition,  the  complicated  roles  and  responsibilities  of  a  joint  program  integrating 
service-led  major  defense  acquisition  programs  (MDAPs),  a  cumbersome  Department  re¬ 
quirements  generation  and  acquisition  process,  international  treaty  constraints,  and  an  in¬ 
consistent  budget  process  affected  the  BMD  system  development  and  the  supporting  M&S 
effort.  The  current  state  of  missile  defense  system  representations  in  the  BMD  System- 
Level  M&S,  discussed  in  this  chapter,  occurred  partly  as  the  result  of  the  many  organiza¬ 
tional  impacts  and  Department  process  constraints. 

Since  the  1980’s  the  Agency’s  models  and  simulation  program  reflected  changing 
National  missile  defense  priorities  evolving  from  the  bi-polar  Cold  War  environment  to 
counter  the  myriad  of  emerging  and  uncertain  threats  of  the  post-Cold  War  world: 

•  1984-1987  -  Explore  technologies  for  national  ballistic  missile  defense, 

•  1987-1991  -  Start  acquisition  of  a  phased  national  ballistic  missile  defense 
(NMD), 

•  1991-1993  -  Acquire  a  limited  global  ballistic  missile  defense  (GPALS), 

•  1993-1996  -  Develop  and  field  a  Theater  Missile  Defense  (TMD).  Continue  NMD 
as  a  technology  readiness  program, 

•  1996-2001  -  Continue  to  acquire  TMD.  Develop  a  NMD  system  for  possible  lim¬ 
ited  deployment  [BMDOOa], 

•  2002-Present  -Develop  and  field  an  integrated  Ballistic  Missile  Defense  System 
(BMDS)  [Rum02c,  Ald02e], 
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At  the  turn  of  the  century  the  Agency  managed  separate  TMD  and  NMD  system  architec- 
tures  “  ,  and  lacked  a  single  unified  missile  defense  system  architecture,  while  the  Agency 
executed  three  concurrent,  and  often-competing  acquisition  strategies: 

•  Field  TMD  systems  quickly, 

•  Determine  the  deployment  strategy  for  the  NMD  systems, 

•  Continue  to  advance  the  technology  [BMDOOa]. 

The  Service  Components  TMD  programs,  and  Joint  Program  office-managed  NMD 
program  developed  their  respective  missile  defense  system  efforts  as  MDAPs,  with  the 
BMDO  responsible  for  integrating  the  diverse,  complex  weapon  and  sensor  systems  into 
single,  seamless  interoperable  operational  system  architecture  [BMDOOd].  The  Department 
designed  the  Army’s  PATRIOT  program  (e.g.,  PAC-2,  PAC-2  GEM,  and  PAC-3)  and 
Navy  Area  Defense  (NAD)  lower-tier"  missile  defense  systems  to  defend  against  terminal 
endoatmospheric  threats.  At  higher  altitudes,  the  Department  devised  the  Army’s  Thea¬ 
ter  Area  Air  Defense  System  (THAAD)  and  Navy  Theater  Wide  (NTW)  missile  defense 
systems'  to  counter  upper-tier  exoatmospheric"  '  missile  threats  [BMDOOd]. 

The  NAD  and  NTW  missile  defense  systems  evolved  from  the  Aegis  Weapon  Sys¬ 
tem’s  primary  mission  as  a  fleet  air  defense  system  [BMD99a].  Other  missile  defense  ef- 
forts  included  the  Airborne  Laser  (ABL)  program,  an  Air  Force  managed  system.  The 
PAC-3,  THAAD,  and  NTW  systems  employed  hit-to-kill  (HTK)235  technology.  Appendix 
A  summarizes  the  mission,  organization,  technical,  acquisition,  management,  procedures 
processes  supporting  the  Agency’s  evolution  and  organizational  responsibilities  in  the 
1996-2001  timeframe. 

The  Agency  also  supported  three  major  international  collaborative  programs  includ¬ 
ing  an  international  cooperative  program  with  Germany  and  Italy,  the  Medium  Extended 
Air  Defense  System  (MEADS)236.  The  Israeli  ARROW  program  [BMDOOj],  now  deployed 
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System  architecture  in  this  context  is  the  software  architecture  plus  execution  and  development  environments. 

The  PAC-3  and  NAD  missile  defense  weapons  employed  against  endoatmospheric  threats  are  referred  to  as  lower-tier  systems. 
Endoatmospheric  refers  to  being  within  the  Earth’s  atmosphere;  generally  considered  tro  be  altitudes  below  100  km  [MDA02b]. 
THAAD  and  NTW  are  also  called  mid-course  systems 

Exoatmospheric  refers  to  being  outside  the  Earth’s  atmosphere;  generally  considered  to  be  altitudes  above  100  km  [MDA02b}. 
The  ABL  concept  employed  a  multimegawatt  chemical  laser  to  destroy  missiles  in  the  boost  phase  [BMDOOa]. 

Hit-to-kill  technology  (HTK)  employs  kinetic  energy  to  destroy  the  target  [BMDOOa]. 

MEADS  was  conceived  to  provide  deployed  NATO  maneuver  forces  with  360  degree  protection  [BMDOOa]. 
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and  operational  is  another  international  missile  defense  effort  developed  with  U.S.  support, 
and  employs  a  high-explosive  warhead  rather  than  the  HTK  technology  designed  into  U.S. 
missile  systems.  The  Russian-American  Observable  Satellite  (RAMOS)  is  a  cooperative 
effort  to  observe  the  earth’s  atmosphere  and  ballistic  missile  launches. 

However,  as  the  United  States  enters  the  21st  century,  the  dynamic  geo-strategic  en¬ 
vironment  no  longer  checked  by  bi-polar  world  power  control  presents  new  challenges. 
New  and  uncertain  21st  century  threats  such  as  missile  attacks  from  rogue  nations,  terror¬ 
ism,  and  weapons  of  mass  destruction  (WMD),  brutally  driven  home  to  the  American  Na¬ 
tion  on  September  11,  2001,  eclipse  the  Cold  War  focus  on  a  massive  Soviet  missile  attack. 

2.  A  MISSION-CRITICAL  SYSTEM  IN  THE  NATIONAL  DEFENSE 

On  January  2,  2002,  the  Secretary  of  Defense  [Rum02c]  established  the  Missile  De- 
fense  Agency  (MDA),  revised  the  agency  concept  of  operations"  ,  and  directed  the  initia¬ 
tion  of  a  single  joint  program  to  develop  an  integrated  ballistic  missile  defense  system 
(BMDS).  [Rum02c]  also  directed  a  capability-based  requirements  process,  supported  by 
the  full  and  cooperative  efforts  of  the  Services,  Joint  Staff,  and  defense  agencies  to  achieve 
the  objectives  of  this  National  priority  (Appendix  B). 

The  Department’s  organizational  change  of  the  BMDO  to  the  MDA  was  part  of  a 
transformation  process  to  replace  an  overly  restrictive,  non-responsive,  and  overly  prescrip¬ 
tive  requirements  and  acquisition  process  [Ald02e].  These  significant  policy  changes  re¬ 
moved  the  communication  barriers  with  the  former  MDAPs,  and  pennitted  a  two-way  dia¬ 
logue  establishing  a  foundation  for  improved  system  representations  in  the  BMDS  System- 
Level  M&S.  However,  the  creation  of  the  BMDS  also  added:  an  expanded  mission,  a 
wider  scope  of  ballistic  missile  defense  responsibility,  many  new  complex  program  re¬ 
quirements,  and  an  accelerated  milestone  schedule  placing  renewed  emphasis  on  the  devel¬ 
opment  of  a  credible  BMDS  M&S  program. 

Nearly  one  year  later  on  December  17,  2002,  President  Bush  identified  the  impor¬ 
tant  role  of  missile  defense  to  the  Nation,  friends,  and  allies,  and  directed  the  Secretary  of 
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[Rum02c]  described  the  new  MDA  organizational  structure,  roles,  responsibilities,  processes,  and  policies  that  detailed  the  way  the 
MDA  operates. 
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Defense  to  “proceed  with  the  fielding  of  an  initial  set  of  missile  defense  capabilities” 
[Bus02f].  In  addition  the  President  made  the  following  statement  underscoring  the  mis¬ 
sion-critical  nature  of  missile  defense: 

The  deployment  of  missile  defenses  is  an  essential  element  in  our  broader 
efforts  to  transform  our  defense  and  deterrence  policies  and  capabilities  to 
meet  the  new  threats  we  face.  Defending  the  American  people  against  these 
new  threats  is  my  highest  priority  as  Commander-in-Chief,  and  the  highest 
priority  of  my  Administration  [Bus02f]. 

[Rum02c]  requires  a  single  missile  defense  program  to  design,  develop,  and  test  the 
BMD  system  elements  (e.g.,  the  former  MDAPS)  of  an  integrated  BMDS  to  defend  the 
U.S.,  deployed  forces,  allies,  and  friends,  with  a  BMDS  that  layers  defenses  to  intercept 
missiles  in  all  phases  of  their  flight  phases  (boost,  midcourse,  and  terminal)  against  all 
ranges  of  threats  [MDA02a,  MDA02c,  MDA02b,  MDA02e,  MDA02f,  MDA02h,  MDA02j, 
MDA02m,  MDA02x,  MDA02y,  MDA03b];  support  the  fielding  of  element  capabilities, 
such  as  PATRIOT  Advanced  Capability-3  (PAC-3)  system,  as  soon  as  practicable;  de¬ 
velop,  test  technologies  and  improve  missile  defense  test  ranges  [Pat99,  TRG01];  and  pro¬ 
vide  early  capability,  if  necessary,  inserting  new  technologies  as  they  become  available,  or 
when  the  threat  warrants  [Bus02e,  Rum02c,  Kad02b,  Kad02c,  MDA02i,  MDA03e].  The 
Under  Secretary  of  Defense  (AT&L)  [Ald03]  noted,  “space  and  missile  defense  is  central 
to  the  future  of  our  national  security”  [Ald03],  reconfirming  missile  defense  as  a  mission- 
critical  requirement  for  the  Nation. 

In  addition  to  the  HTK  interceptor  technology  and  directed  energy  weapons,  the 
Agency  is  also  developing  sensor  systems  that  will  improve  the  ability  to  detect,  track,  and 
identify  ballistic  missile  warheads  from  countermeasures,  which  will  be  integrated  in  the 
BMDS  through  a  Battle  Management  /  Command  and  Control  (BM/C2)  system.  The 
Agency  is  also  exploring  advanced  weapon  capabilities  including  space-based  lasers,  and 
sea-  and  space-based  kinetic  energy  systems  [MDA02p], 

Figure  9-2  illustrates  the  major  BMDS  functions  required  of  an  integrated  collection 
of  defense-in  depth  capabilities  (e.g.,  detection,  identification,  classification,  battle  man¬ 
agement,  sustainment,  engagement,  kill  assessment)  supporting  the  Boost  Phase  Defense 
Segment  (BDS),  Midcourse  Defense  Segment  (MDS),  and  a  Tenninal  Defense  Segment 
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(TDS)  (e.g.,  the  missile  defense  kill-chain).  A  planned  Missile  Defense  Test  Bed  in  the 
Pacific  will  support  these  three  segments  with  additional  test  realism,  the  ability  to  support 
multiple  engagements,  and  provide  a  limited  contingency  capability.  Appendix  B  provides 
a  synopsis  of  U.S.  missile  defense  acquisition  programs,  international  collaborative  efforts, 
technology  programs,  and  the  BM/C2,  weapon,  sensor,  and  communication  components  of 
these  programs  in  2002. 


M  idcourse  Phase 
Defense  Segment 


Figure  9-2.  BMDS  Defense  Phase  Segments  and  Required  Capabilities  (After  [RayOlc]) 

3.  A  HIGH-RISK,  SOFTWARE-INTENSIVE  SYSTEM 

Missile  defense  has  other  hurdles  including  the  software  challenge  identified  by 
Lieutenant  General  George  Monahan  for  the  SDI  program  in  his  1990  report  to  the  Secre¬ 
tary  of  Defense  when  he  noted, 

The  greatest  engineering,  vice  technical,  challenge  in  the  SDI  program  is 
software. . .  [BMDOOa]. 

BMDS  software-intensive  systems  will  be  complex,  expected  to  support  system  evolution, 
perform  the  most  difficult  tasks,  including  battle  management,  recover  from  software  and 
hardware  failures,  and  respond  correctly  to  anticipated  and  unanticipated  threats  to  the  sys¬ 
tem.  Critics  and  supporters  agree  that  software  errors  will  occur  in  a  system  as  complex  as 


223 


the  BMD,  but  argue  over  whether  the  failures  would  be  catastrophic211'  [Bow02],  [ParOl] 
believes  the  development  of  dependable,  trustworthy  software-intensive  systems  for  the 
BMDS  is  a  very  high-risk  undertaking  and  may  very  well  be  impossible. 

Correctly  assessing  and  meeting  the  many  challenges  for  developing  quality  soft¬ 
ware-intensive  systems  supporting  systems  engineering/integration,  BMD  System  testing, 
operations,  and  simulation  development  with  credible  verified  and  validated  software  prod¬ 
ucts  is  on  the  critical  path  to  fielding  an  integrated  missile  defense  system.  In  addition  to 
supporting  the  evolutionary  acquisition  of  the  BMDS  program  and  fielding  of  block  capa¬ 
bilities  every  two  years,  the  evolving  BMD  M&S  program  will  also  support  all  missile  de¬ 
fense  life  cycle  support  requirements. 

D.  BMD  SYSTEM  M&S  DOMAIN  OVERVIEW 

The  Agency  M&S  program  [MDA03M]  supporting  the  development  of  the  BMDS 
evolved  from  the  large-scale,  legacy  M&S  development  efforts  of  previous  missile  defense 
programs  [SCC+88,  SW96].  The  current  M&S  programs  has  three  categories  mapped  to 
the  Department’s  M&S  Hierarchy  in  Figure  2-2.  The  three  categories  of  the  Agency’s 
four-level  M&S  Hierarchy  illustrated  in  Figure  9-3  are  the:  BMD  System  Threat,  Signa¬ 
ture,  Environment,  and  Lethality  (TSEL).  The  BMD  System  TSEL  M&S,  at  the  bottom 
layer  of  the  Agency’s  four-level  hierarchy  support  the  development  of  the  BMD  Element- 
Level  components/sub-systems/systems,  at  the  second  layer  of  the  hierarchy.  The  BMD 
Element-level  M&S  directly  support  the  Element  programs.  The  BMD  System-Level 
M&S  at  the  third  level  of  the  hierarchy  M&S  supports  the  system-wide  integration  and  in¬ 
teroperability  of  the  element  representations  into  the  BMD  system. 

The  BMD  System-Level  M&S  and  the  BMD  System  TSEL  comprise  the  BMDS 
Core  Model  set.  The  BMD  System-Level  M&S  is  the  only  category  addressed  in  detail 
within  this  dissertation,  while  the  Element-Level  M&S,  and  BMD  System  Threat,  Signa¬ 
ture,  Environment,  and  Lethality  M&S  are  summarized  at  a  very  high  level.  See  Chapter 


Several  meanings  of  catastrophic  have  been  developed,  but  [SCC+88]  defined  a  catastrophic  failure  as  a  decline  in  system  perform¬ 
ance  to  a  10  percent  or  less  of  expected  performance. 
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VII  for  a  summary  of  the  Department’s  Joint  simulation  development  program,  (e.g., 
JWARS  JSIM,  and  JMASS),  the  Agency’s  capstone  M&S  category  shown  in  Figure  9-3. 


Figure  9-3.  BMD  M&S  Addressed  in  this  Study  (After  [PMR97]) 

1.  BMDS  Legacy  Model  and  Simulation  Systems 

A  major  MDA  objective  is  to  provide  credible  M&S  and  improve  the  Department’s 
confidence  in  BMDS  simulation  results  [Kad02a,  Kad02b,  Kad02c].  However,  the  BMDS 
M&S  program  has  four  major  challenges  to  overcome  in  support  of  the  vision  to  make  mis¬ 
sile  defense  a  reality:  1)  developing  credibility  in  the  M&S  and  building  user  confidence  in 
the  results,  2)  accurately  modeling  the  significant  technological  hurdles  inherent  in  missile 
defense,  3)  dealing  with  major  acquisition  program  complexity,  and  4)  the  software  engi¬ 
neering  challenge  itself. 

The  Agency  and  its  predecessor  Service  /  Agency  /  Component  organizations  de¬ 
veloped  an  expensive  portfolio  of  M&S  including  large-scale,  legacy  simulation  systems: 

Adage,  Cannonette,  and  COMO  III  [RBB+82,  Che87]  to  support  the  air  and  missile  de- 
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fense  mission.  Many  of  these  M&S  evolved  in  an  ad  hoc  manner,  evolving  into  large  sys¬ 
tems  with  multiple  stakeholders  and  significant  support  infrastructures.  A  significant  num¬ 
ber  of  the  existing  systems  lacked  fonnal  documented  conceptual  models  and  reflected  an 
inconsistent  V&V  history  [RRB+82,  Che87]. 

In  addition,  the  number  of  M&S  tools  in  the  missile  defense  domain  proliferated  in 
the  1990’s  until  there  were  nearly  three  hundred  M&S  systems  competing  for  BMDO  M&S 
life  cycle  funding  and  support  [BMD99b].  The  Agency’s  legacy  M&S  systems  supported 
experiments,  analysis,  studies,  and  tests  with  little  documented  V&V,  resulting  in  reduced 
credibility  of  the  accreditation  process  for  these  simulations,  and  limiting  confidence  in  ex¬ 
periment,  analysis,  study,  or  test  results  [RRB+82,  Che87]. 

In  a  series  of  agency-level  M&S  reviews  conducted  during  1998  and  1999,  redun¬ 
dant,  low-use  M&S,  and  simulation  systems  with  a  low  potential  for  future  integration  or 
HLA  interoperability  were  discontinued,  while  approximately  eighty-eight  M&S  tools  were 

O'XQ 

determined  to  be  MDAP -unique'  and  management  responsibilities  were  assigned  to  the 
fonner  major  defense  acquisition  programs  (e.g.,  PATRIOT,  THAAD)  [BMD99b].  The 
MDAP -unique  tools  currently  constitute  the  MDA  Element-Level  M&S  category  of  the 
MDA  M&S  Hierarchy  in  Figure  9-3.  The  remaining  common-,  and  general-use  legacy 
tools,  supporting  multiple  internal  and  external  MDA  stakeholders,  employed  in  all  phases 
of  the  BMD  system  evolutionary  program  development  (e.g.,  RDT&E,  Transition,  Opera¬ 
tions  &  Maintenance  phases),  and  integrated  into  multiple  functional  areas  (e.g.,  analysis, 
training,  experimentation,  acquisition,  and  operations),  became  the  BMDS  Core  M&S. 
Figure  9-3  illustrates  the  two  categories  of  the  BMDS  Core  M&S:  the  BMDS  Threat,  Sig¬ 
natures,  Environment,  Lethality  and  Threat  (TSEL)  category  and  the  BMDS  System-Level 
M&S. 


2.  BMDS  System-Level  Legacy  M&S  Systems 

The  five  simulations  comprising  the  MDA  System-Level  M&S  layer  of  the  MDA 
M&S  Hierarchy  [SW96]  shown  in  Figure  9-3  comprise  the  scope  of  this  research.  The  sys¬ 
tems  under  study  are:  the  Commanders  Analysis  and  Planning  Simulation  (CAPS),  the 
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MDAP-unique  M&S  were  determined  to  be  valid  requirements  for  an  individual  MDAP,  but  did  not  have  system-wide  applicability. 
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Missile  Defense  Wargame  and  Analysis  Repository  (MDWAR),  the  Extended  Air  Defense 
Simulation  (EADSIM),  the  Extended  Air  Defense  Testbed  (EADTB),  and  the  Missile  De¬ 
fense  System  Exerciser  (MDSE).  Theoretically,  the  BMD  System-Level  M&S  in  this  layer 
include  representations  of  the  Element-Level  systems  at  varying  degrees  of  fidelity,  accu¬ 
racy,  precision,  and  resolution.  In  the  future,  the  BMD  System-Level  M&S  will  in  turn 
support  development  of  valid  BMDS  representations  in  the  Department’s  capstone-level 
joint  M&S. 

The  BMD  System-Level  M&S  support  environments240  are  repositories  of  missile 
defense  domain  expertise,  critical  intellectual  property,  and  the  foundation  for  the  devel¬ 
opment  of  future  components  in  the  missile  defense  domain  core  asset  portfolio.  This  con¬ 
stituency  of  knowledgeable  stakeholder  supporters  is  almost  always  lacking  in  the  devel¬ 
opment  of  new  systems,  where  requirement-stakeholders  often  compete  for  priority,  fund¬ 
ing,  and  allocation  of  development  resources.  The  MDA  also  supports  external  stake¬ 
holders  including  the  Department’s  operational  test  community,  National-level  decision¬ 
makers,  and  international  cooperative  partners. 

a.  Commanders  Analysis  and  Planning  Simulation  (CAPS) 

The  CAPS  model  is  a  theater  level,  force-on-force,  missile  defense  system 
analysis  tool,  and  qualitatively  considered  low-fidelity.  CAPS  simulates  active  theater  de¬ 
fense  against  ballistic  missiles  and  air  breathing  threats  and  is  employed  as  a  planning, 
training  and  analysis  tool  with  four  views:  footprint  view,  operating  area  view,  defended 
area  view,  and  scenario  view.  These  capabilities  illustrate  defensive  footprints,  or  areas  on 
the  ground  that  may  be  protected  by  missile  defense  units,  possible  operating  regions  for 
defensive  systems,  areas  defended  by  various  positioning  of  defensive  forces,  and  expected 
performance  of  missile  defense  systems  in  a  campaign.  See  Appendix  C  for  additional  in¬ 
formation. 
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The  support  environment  includes  the  multiple  stakeholders,  both  government  and  private  sector,  who  manage,  develop,  test,  V&V, 
use,  or  support  the  M&S  development  process. 
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b.  Missile  Defense  Wargame  and  Analysis  Resource  (MDWAR) 

The  MDWAR  simulation,  formerly  called  WARGAME  2000,  is  a  consid¬ 
ered  a  low-medium  fidelity  simulation  designed  to  provide  the  missile  defense  community 
with  a  Human-In-Control  (HIC)241  wargaming  capability.  MDWAR  provides  a  construc¬ 
tive  M&S  tool  to  develop  missile  defense  concepts  of  operation  CONOPS),  doctrine,  tac¬ 
tics,  techniques  and  procedures242  (TTPs)  through  the  use  of  virtual  experimentation  in  a 
synthetic  environment.  MDWAR  is  the  successor  system  to  ARGUS  simulation,  devel¬ 
oped  to  provide  a  real-time,  discrete-event  simulation  capability  supporting  the  develop¬ 
ment  of  missile  defense  [TOM99].  MDWAR  also  supports  missile  defense  architecture 
evaluation,  joint  missile  defense  exercises,  and  the  development  of  the  BMDS  BM/C2  ele¬ 
ment. 

MDWAR  HIC  experiments  interactively  simulate  the  complex  TMD  and 
NMD  threat  environment  with  scenarios  designed  to  induce  the  effects  the  real  world  con¬ 
fusion  to  the  operators  manning  realistic  operator  consoles  and  displays.  The  objectives  of 
the  MDWAR  war  games  are  to  exercise  command  and  control  (C2)  processes  in  a  HIC  en¬ 
vironment  with  realistic  battle  management  defense  system  features  that  enable  the  opera¬ 
tors  to  perceive  and  interpret  the  battlefield  situation,  identify  problems,  develop  courses  of 
action,  and  allows  operators  to  select,  implement  and  monitor  the  corrective  action  and  its 
impact  on  the  situation.  See  Appendix  D  for  additional  infonnation. 

c.  Extended  Air  Defense  Simulation  (EADSIM) 

EADSIM  is  a  constructive  simulation  first  released  in  1989  [JAS97c]  as  a 
system-level  simulation  providing  a  many-on-many  theater-level  simulation  of  air,  space, 
and  missile  warfare.  EADSIM  is  generally  considered  a  medium  fidelity  simulation. 
EADSIM  models  joint  and  combined  force  air  and  missile  defense  warfare,  ranging  from 
few-on-few  to  many-on-many  theater  scenarios,  to  detennine  the  effectiveness  of  air  and 
missile  defenses  against  the  full  spectrum  of  threats  as  an  analysis  tool  and  to  augment 
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Also  referred  to  as  human-in-the-loop  (HIL) 

242  Computer-based  TTP,  CONOPS,  and  doctrine  development  supports  the  force  and  combat  development  functions  of  the  services  and 
joint  forces  and  provides  an  early  opportunity  for  the  war  fighter  to  influence  the  system  development  and  employment.  TTPs  are  also 
reviewed  early  in  the  material  development  process  to  see  if  a  change  to  TTPs  CONOPS,  or  doctrine  negated  the  need  for  a  materiel 
solution. 
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training.  EADSIM  supports  the  four  pillars  of  theater  missile  defense:  active  defense,  pas¬ 
sive,  defense,  attack  operations,  and  BM/C2.  EADSIM  also  models  the  command  and  con¬ 
trol  (C2)  decision  processes,  intelligence  gathering,  and  the  inter-platform  communication 
process.  See  Appendix  E  for  additional  information. 

d.  Extended  Air  Defense  Testbed  (EADTB) 

The  EADTB  simulation,  an  event-driven,  Monte  Carlo  constructive  simula¬ 
tion  framework,  supports  missile  defense  modeling  from  the  fire -unit  to  the  theater  level 
[BMD01],  EADTB  evaluates  the  effectiveness  and  efficiency  of  weapon  systems  against 
targets  and  the  value  of  different  force  structures  and  resources.  The  EADTB  system  de- 
sign  offers  object-based"  simulation  architecture  with  the  flexibility  to  use  high-  or  low- 
detail,  user-developed,  specific  system  representations  (SSRs)244,  for  play  on  a  simulated 
game  board.  [SMP98,  Ray02a].  EADTB,  considered  medium-to-high  fidelity  is  capable  of 
supporting  a  wide  range  in  the  level  of  SSR  detail  in  a  single  simulation  exercise,  running 
interactively  or  in  batch  mode,  and  provides  analytical  flexibility,  with  the  objective  of  re¬ 
ducing  the  need  for  multiple  simulations  [BBB+96].  See  Appendix  F  for  additional  infor¬ 
mation. 


e.  Missile  Defense  System  Exerciser  (MDSE) 

MDSE,  originally  called  the  Theater  Missile  Defense  System  Exerciser 
(TMDSE)  [Too94]  addressed  the  objectives,  architecture  and  engineering  considerations  of 
the  theater  level  missile  defense  System-Level  hardware-in-the-loop  capability  [BB98a, 
BB98b].  MDSE  is  the  highest  fidelity  BMD  System-level  simulation.  MDSE  was  origi¬ 
nally  designed  to  demonstrate  the  interoperability  of  theater  ballistic  missile  defense 
(TBMD)  weapon  systems  and  national  sensor  elements  in  a  real-time,  GPS-synchronized, 
centrally  controlled,  geographically  distributed  hardware-in-the-loop  (HWIL)  tool  to  test 
the  TBMD  family  of  systems  (FoS)  in  a  dynamic  environment  [Too94].  The  original 
MDSE  development  concept  specified  by  [Too94]  identified  the  requirements  for  the  first 
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EADTB  is  object-based,  versus  object  oriented,  since  it  does  not  support  inheritance. 

244 

SSRs  are  data  and  code  rule  sets  developed  to  represent  objects  (THAAD,  PATRIOT)  and  control  their  behavior  [BBB+96], 
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three  TMDSE  builds  and  called  for  an  incremental  build  schedule  to  develop  a  test  resource 
supporting  interoperability  and  integration  tests  of  the  theater  missile  defense  architecture. 
The  GMD  (formerly  NMD)  architecture  was  not  a  part  of  the  original  TMDSE  require¬ 
ments,  and  will  be  addressed  in  future  MDSE  build  requirements.  See  Appendix  G  for  ad¬ 
ditional  information. 

f  BMD  System-Level  M&S  Status 

These  simulations  experienced  different  growth  patterns.  In  the  case  of 
CAPS  and  EADSIM  the  documented  start  dates  for  simulation  development  do  not  reflect 
earlier  development  efforts.  For  instance,  CAPS  evolved  from  the  Operation  Planning 
Simulation  (OPS)  [BBG+99f],  while  EADSIM  built  on  the  foundation  of  the  earlier 
C3ISIM  simulation  [BBG+99e].  In  comparison  to  the  CAPS  and  EADSIM  evolutionary 
development,  MDWAR  and  MDSE  were  more  deliberate  designs.  The  five  BMD  System- 
Level  simulations  under  study  are  currently  in  the  maintenance  phase  of  the  M&S  software 
life  cycle  and  executing  to  a  stable  program  plan245. 

E.  BMD  SYSTEM-LEVEL  SIMULATION  MODEL  REPRESENTATIONS 

This  section  provides  the  analysis  results  illustrating  the  status  of  acceptable  BMD 
System  representations  within  the  five  BMDS  System-Level  M&S  (e.g.,  CAPS,  MDWAR, 
EADSIM,  EADTB,  and  MDSE).  The  analysis  includes  the  following  missile  defense  rep¬ 
resentations: 

•  Missile  Defense  Weapons  -  Tenninal  Defense  Systems  (TDS),  Midcourse  Defense 
Systems  (MDS),  and  International, 

•  Directed  Energy  Weapons  -  Airborne-  and  Space-based  Boost  Defense  Systems 
(BDS), 

•  Kinetic  Energy  Weapons  -  Sea-  and  Space-based  Boost  Defense  Systems  (BDS), 

•  Radar  Sensors  -  Tenninal,  Midcourse,  and  International  systems, 

•  Satellite  /  Infrared  Sensors, 

•  Airborne  Laser  Sensors, 

•  Platfonn  Systems 

•  Geo-Spatial  Environment  representations. 


The  current  program  plan  for  the  M&S  under  study  shows  a  steady  state  program  of  resources,  factoring  in  inflation,  for  most  of  the 
decade.  Any  significant  decrements  to  the  projected  program  will  extend  the  time  needed  to  improve  the  credibility  of  the  BMDS  M&S. 
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For  the  purpose  of  this  research  an  acceptable  authoritative  representation  met  the 
following  criteria:  the  appropriate  program  office  previously  accredited  the  representation 
for  a  specific  use,  or  the  program  office  developed  and  documented  the  representation  with 
an  understanding  of  the  representations’  planned  intended  use,  or  the  program  office  re¬ 
viewed  and  accepted  the  system  representation,  with  specific  caveats  for  some  documented 
intended  use.  Documentation  supporting  the  PAC-3  Initial  Operational  Test  and  Evalua¬ 
tion  (IOT&E)  Interoperability  Demonstration  (PHD)  [CRC02a,  Bar02a,  CRC02b,  Bar02b] 
conducted  in  April  2002  proved  invaluable  for  the  purpose  of  assessing  representation  ac¬ 
ceptability.  Based  on  this  criteria,  a  “Y”  in  the  following  tables  indicates  evidence  of  at 
least  one  Acceptable  System  Representation,  and  an  “N”  indicates  an  Unacceptable  /  Miss¬ 
ing  System  Representation.  These  statistics  do  not  state,  imply  or  suggest  a  blanket  ap¬ 
proval  of  system  representation  certification,  accreditation,  or  validation  by  the  respective 
MDA  element  program  office  for  any  future  uses. 

1.  BMD  System-Level  M&S  TDS  Weapon  System  Representations 

In  the  current  TDS  population  of  six  missile  systems  identified  in  Table  9-1,  five 
systems  (83%)  have  acceptable  representations  of  the  specific  missile  in  the  BMD  System- 
Level  core  simulations.  One  specific  representation  configuration  of  the  THAAD  EMD 
missile  system  is  currently  available  at  a  very  high  level  of  aggregation  in  CAPS.  Overall, 
eighty-seven  percent  (26/30)  of  the  TDS  missile  representations  are  included  in  the  BMD 
System-Level  M&S.  This  is  the  highest  percentage  of  weapon  system  representations 
found  in  the  BMD  System-Level  M&S.  Only  the  TDS  sensor  system  representations,  dis¬ 
cussed  later  in  the  chapter,  exceed  this  metric  in  the  System-Level  models.  The  relatively 
high  percentage  of  missile  representations  within  the  TDS  may  be  attributed  to  the  follow¬ 
ing: 

•  A  focus  on  tenninal-phase  systems  (PATRIOT,  THAAD,  Aegis)  during  the  devel¬ 
opment  phase  of  the  simulations, 

•  An  emphasis  on  interoperability  among  the  different  terminal-phase  systems, 

•  The  requirement  for  a  Single  Integrated  Air  Picture  (SIAP)  including  theater  ballis¬ 
tic  missiles,  cruise  missiles,  and  aircraft,  both  enemy  and  friendly, 
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•  A  missile  defense  program  priority  during  most  of  the  1990s  to  defend  U.S.  forces 
and  allies  in  a  theater-wide  operational  scenario, 

•  The  PATRIOT  and  the  Aegis  terminal  defense  weapon  systems  were  the  most  ma¬ 
ture  missile  defense  systems, 

•  A  higher  probability  of  M&S  reuse  within  a  mature  program  office  with  a  family  of 
related  systems,  such  as  the  PATRIOT  program  office, 

•  Support  contractor  longevity  and  domain  expertise  for  different  generations  of  simi¬ 
lar  missile  system  (e.g.,  PAC-2,  PAC-2  GEM,  PAC-3)  [BRC+01]. 

Of  significance,  a  specific  system  representation  based  on  correct  specifications  for 
the  system  at  certain  stage  of  the  development  cycle,  even  with  an  accurate  verified  and 
validated  conceptual  model  may  be  completely  invalid  for  another  intended  use.  Table  9-1, 
illustrates  this  apparent  inconsistency  of  specific  system  representations  with  the  availabil¬ 
ity  of  the  THAAD  Program  Definition  and  Risk  Reduction  (PDRR)246  phase  missile  system 
version  and  the  absence  of  the  THAAD  Engineering  and  Manufacturing  (EMD“  )  phase 
missile  system  within  the  BMD  System-Level  simulation  systems. 

Missing  or  inaccurate  system  characteristics  or  critical  attributes  [MDW+01]  may 
adversely  impact  interoperability  among  the  federation  entities  resulting  in  substantive  in¬ 
teroperability  issues  including  representational  accuracy  and  internal  sensitivity  problems. 
There  are  several  plausible  reasons  for  the  situation  including  a  lack  of  coordination  for  a 
federation  conceptual  model,  inadequate  requirement  management,  poor  configuration 
management,  lack  of  system  information,  non-responsive  contract  support,  funding  short¬ 
falls,  and  scheduling.  Whatever  the  root  cause,  the  lack  of  specific  valid  system  representa¬ 
tions  may  adversely  impact  the  simulation  or  federations  intended  use  and  /  or  provide  de¬ 
cision-makers  with  inaccurate  simulation  results. 


BMDS  TDS  MISSILE  SYSTEM  WEAPON  REPRESENTATIONS 

TDS  MISSILES 

CAPS 

MDWAR 

EADSIM 

EADTB 

MDSE 

PAC-2 

Y 

Y 

Y 

Y 

Y 

PAC-2  GEM 

Y 

Y 

Y 

Y 

Y 

PAC-3 

Y 

Y 

Y 

Y 

Y 

THAAD  PDRR 

Y 

Y 

Y 

Y 

Y 
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PDRR  is  the  early  Program  Definition  and  Risk-Reduction  Phase  of  a  system’s  development. 
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EMD  systems  have  progressed  beyond  PDRR,  with  more  mature  engineering  specifications  supporting  manufacturing  of  the  system. 
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THAAD  EMD 

Y 

N 

N 

N 

N 

AEGIS  SM-2  BLK IV A248 

Y 

Y 

Y 

Y 

Y 

#  Avail  /Total/  %  Avail 

6  /  6  / 100% 

5/6/83% 

5/6/83% 

5  /  6  /  83% 

5/6/83% 

[SpaOO, 

[CM99, 

[BAC+95, 

[BBG+96, 

[Too94, 

SpaOl, 

TOM99, 

TOM97, 

RayOla, 

McQ97, 

Citation  from 

BBG+99f[ 

TRWOOa, 

BBG+99e, 

Rayo2a, 

TEM00] 

TRWOOb, 

TBE02a, 

RayOlb] 

TRW02b] 

TBE02b] 

Table  9-1. 


erminal  Defense  Segment  Missile  Representations 


The  BMDS  System-level  M&S  program  also  supports  two  significant  international 
simulation  development  activities  cited  in  Table  9-2,  and  an  international  collaborative 
M&S  program  detailed  in  Tale  9-17.  Systems  such  as  the  Israeli  ARROW  Weapon  System 
are  deployed  operational  systems  with  real-world  interoperability  requirements,  while 
MEADS  is  a  joint  cooperative  German,  Italian,  and  U.S.  missile  defense  development  pro¬ 
gram. 


BMDS  INTERNATIONAL  TDS  MISSILE  WEAPON  SYSTEM  REPRESENTATIONS 

INTL  MISSILES 

CAPS 

MDWAR 

EADSIM 

EADTB 

MDSE 

ARROW  (ISRAEL) 

Y 

Y 

Y 

Y 

Y 

MEADS  (GE,  IT,  US) 

N 

Y 

N 

N 

N 

1  /  2  /  50% 

1121 100% 

1  /  2  /  50% 

1  /  2  /  50% 

1  /  2  /  50% 

[SpaOO, 

[CM99, 

[BAC+95, 

[BBG+96, 

[Too94, 

SpaOl, 

TOM99, 

TOM97, 

RayOla, 

McQ97, 

Citation  from 

BBG+99f[ 

TRWOOa, 

BBG+99e, 

Rayo2a, 

TEM00] 

TRWOOb, 

TBE02a, 

RayOlb] 

TRW02b] 

TBE02b] 

Table  9-2.  International  Terminal  Defense  Segment  Missile  Representations 


2.  BMDS  System-Level  M&S  MDS  Weapon  System  Representations 

The  MDS  missile  system  weapon  representations  listed  in  Table  9-3  are  under¬ 
represented  in  the  BMD  System-Level  M&S.  There  are  only  four  weapon  system 
representations  of  the  GMD  ground-based  interceptor  (GBI)  and  one  Navy  Leap  ALI 
missile  system,  based  on  parameter  files,  resident  in  the  BMD  System-Level  M&S.  CAPS, 


248 

All  AEGIS  SM-2,  weapon  systems  in  the  Navy  Area  Defense  (NAD)  program  in  2001  were  counted  as  TDS  representations.  With 
the  evolution  of  the  Navy  Theater  Wide  (NTW)  program  into  the  midcourse  defense  segment  (MDS)  the  AEGIS  SM-3  and  LEAP  ALI 
were  counted  in  the  MDS  category  for  this  study. 
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MDWAR,  and  EADSIM,  three  of  the  relatively  aggregated  M&S  include  the  Aegis  SM-3 
missile.  However,  acceptable  SM-3  missile  representations  are  not  yet  available  in  the 
EADTB  constructive  simulations  or  in  the  MDSE  hardware  in-the-loop  tool.  Reasons  for 
the  lack  of  MDS  missile  representations  in  the  BMD  System-Level  M&S  include  the 
following: 

•  National-level  mission  defense  priorities  shifted  several  times  during  the  1990s  per¬ 
turbing  the  acquisition  strategies,  funding  levels,  and  priority  of  the  different  sys¬ 
tems  and  their  associated  M&S  programs, 

•  The  GMD  (former  NMD)  program  was  developed  as  a  single  joint  program  with  a 
specific  mission  versus  the  family  of  systems  (FoS)  approach  employed  for  theater 
level  missile  defense  systems  requiring  interoperability, 

•  The  GMD/NMD  mission  was  to  defend  the  U.S.  against  ICBM-class  missile 
threats,  as  opposed  to  the  FoS  TAMD  /  TMD  /  TBMD  mission  against  shorter  range 
missile,  aircraft,  and  cruise  missile  threats  in  a  theater  of  operations, 

•  The  GMD/NMD  architecture  was  not  required  or  designed  to  be  interoperable  with 
the  FoS  TAMD  /  TMD  /  TBMD  architecture, 

•  With  the  exception  of  MDWAR,  the  GMD/NMD  missile  representations  were  not 
included  in  the  development  of  the  legacy  BMD  System-Level  M&S, 

•  The  GMD/NMD  program  developed  the  BMD  System-Level  M&S  equivalent  in¬ 
corporating  model  representations  of  the  former  NMD  component  systems  (e.g., 
UEWRS,  DSP,  BM/C2)  in  the  GMD/NMD  system  architecture 

•  Overall,  Table  9-3  shows  only  forty  percent  (8/20)  of  the  MDS  missile  system 
weapon  representations  are  included  in  the  BMD  System-Level  M&S,  with  no  ac¬ 
ceptable  MDS  missile  system  representations  available  in  EADTB  or  MDSE. 


BMDS  MDS  MISSILE  SYSTEM  WEAPON  REPRESENTATIONS 

MDS  MISSILES 

CAPS 

MDWAR 

EADSIM 

EADTB 

MDSE 

GMD  GBI  I  W/EKV 

Y 

N 

N 

N 

GMD  GBI  II  W/EKV 

Y 

Y 

N 

N 

N 

AEGIS  SM-3  BLK  1A 

Y 

Y 

Y 

N 

N 

NAVY  LEAP  ALI 

Y 

N 

N 

N 

N 

#  Avail  /Total/  %  Avail 

4  /  4  / 100% 

3  /  4  /  75% 

1  /  4  /  25% 

0  /  4  /  0% 

0  /  4  /  0% 
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The  GBI  model  is  a  combination  of  fly  out  tables  and  Keplerian  propagation  with  the  boost  phase  GB1  modeled  as  a  unitary  or  single 
stage  interceptor  defined  by  a  two-dimension  fly  out  table.  Interceptor  burnout  conditions  are  extracted  from  the  flyout  tables  and  modi¬ 
fied  to  compensate  the  for  earth’s  influence.  A  point-in-space  navigation  method  is  employed  because  the  two-dimensional  fly  out  tables 
cannot  represent  trajectory  shaping.  Constraints  include  endgame  maneuver  and  the  interceptor  seeker  is  not  explicitly  modeled,  al¬ 
though  a  field-of-view  is  calculated.  Intercept  occurs  at  the  time  of  closest  proximity  to  the  target  with  intercept  outcomes  modeled  with 
two  probabilistic  draws:  probability  of  hit  (Ph)  and  probability  of  kill  (Pk),  and  do  not  represent  complex  functions  of  intercept  geometry, 
velocity,  or  closing  angle  [TOM99], 
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[SpaOO, 

[CM99, 

[BAC+95, 

[BBG+96, 

[Too94, 

SpaOl, 

TOM99, 

TOM97, 

RayOla, 

McQ97, 

Citation  from 

BBG+99f] 

TRWOOa, 

BBG+99e, 

Rayo2a, 

TEM00] 

TRWOOb, 

TBE02a, 

RayOlb] 

TRW02b] 

TBE02b] 

Table  9-3.  BMDS  Midcourse  Defense  Segment  Missile  Representations 


Major  missile  defense  weapon  system  platforms  (e.g.,  Aegis)  must  also  accommo¬ 
date  multiple  configurations  of  weapons,  sensors,  battle  management,  or  communication 
systems  during  the  same  timeframe.  This  is  an  example  of  real  world  interoperability  chal¬ 
lenges.  It  is  also  potential  root  cause  for  substantive  interoperability  issues  in  federations 
involving  the  real-world  AEGIS  SM-3  BLK  1A  and  NAVY  LEAP  ALI  systems  and  their 
respective  system  representations  in  CAPS,  MDWAR,  and  EADSIM  illustrated  in  Table  9- 
3.  In  the  example  highlighted  in  Table  9-3,  there  is  the  potential  for  inadvertently  introduc¬ 
ing  the  incorrect  or  “almost  good”  enough  missile  weapon  system  representation  into  a  fed¬ 
eration,  with  a  potential  to  produce  results  similar  to  the  Ariane  5  mishap. 

3.  BMD  System-Level  M&S  BDS  Weapon  System  Representations 

Supporting  the  Department’s  transformation  process,  evolutionary  acquisition  strat¬ 
egy,  and  an  overall  reduction  in  product  cycle  time,  the  Department’s  M&S  programs  must 
also  transfonn  and  reduce  simulation  development  cycle  time,  especially  for  new  and  pro¬ 
posed  systems  such  as  the  BDS.  Under  the  Block  Development  concept  supporting  the 
fielding  of  missile  defense  capability  by  blocks  in  two-year  intervals  (e.g.,  Block  2004, 
Block  2006. .  .Block  2016)  there  is  a  requirement  to  support  the  system  engineering  process 
with  an  M&S  capability  providing  credible  results  across  the  temporal  spectrum.  This 
would  suggest  that  as  the  Department  system  engineering  process  and  M&S  processes  ma¬ 
ture  together,  the  system  engineer  would  allocate  future  M&S  requirements,  including  fi¬ 
delity,  resolution,  precision,  accuracy  attributes  [MDW+01]  to  future  blocks  and  the  simu¬ 
lation  community  would  respond  with  quality  products  supporting  credibility  and  confi¬ 
dence. 
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The  next  category  of  BMDS  model  representations  is  the  BDS  Directed  Energy 
Weapon  system  category.  In  the  BMD  system  boost  defense  segment  today  there  is  a 
shortfall  of  BDS  directed  energy  (Table  9-4)  and  kinetic  energy  (Table  9-5)  weapon  system 
representations,  with  the  exception  of  the  Airborne  Laser  (ABL)  representations.  The 
PLASTR  and  ISAAC  models  represent  the  ABL  system  in  CAPS  [BBG+99e],  while  the 
ABL  representation  in  MDWAR  is  instantiated  through  a  DIS  gateway  with  the  ABL  op- 
erator-in-the  loop  simulation  residing  at  the  Theater  Air  Command  and  Control  Simulation 
Lacility.  Overall,  only  thirty-three  percent  of  BDS  directed  energy  system  representations 
are  currently  available  in  the  BMD  System-Level  M&S. 


BMDS  BDS  DIRECTED  ENERGY  WEAPON  SYSTEM  REPRESENTATIONS 

DIRECTED  ENERGY 

CAPS 

MDWAR 

EADSIM 

EADTB 

MDSE 

AIRBORNE  LASER 

Y 

Y 

Y 

N 

N 

SPACE-BASE  LASER 

N 

N 

N 

N 

N 

#  Avail  /Total/  %  Avail 

1  /  2  /  50% 

1  /  2  /  50% 

1  /  2  /  50% 

0  /  2  /  0% 

0  /  2  /  0% 

[SpaOO, 

[CM99, 

[BAC+95, 

[BBG+96, 

[Too94, 

SpaOl, 

TOM99, 

TOM97, 

RayOla, 

McQ97, 

Citation  from 

BBG+99f] 

TRWOOa, 

BBG+99e, 

Rayo2a, 

TEM00] 

TRWOOb, 

TBE02a, 

RayOlb] 

TRW02b] 

TBE02b] 

Table  9-4.  Boost  Defense  Segment  Laser  Representations 


The  kinetic  energy  systems  envisioned  for  the  future  of  missile  defense  and  listed  in 
Table  9-5  have  no  acceptable  representations  available,  and  are  maybe  the  most  challeng¬ 
ing  for  the  unprecedented  missile  defense  system  development  effort.  Unlike  the  earlier 
TDS  and  MDS  systems,  which  built  on  previous  legacy  missile  defense  systems,  sub¬ 
systems,  and  domain  bodies  of  knowledge,  future  system  capability  development  faces 
many  unknowns.  Conversely,  since  the  future  may  hold  fewer  constraints  from  current 
stakeholders  and  provide  more  trade  space  flexibility,  the  system  engineer  has  the  potential 
to  evolve  the  system  into  areas  with  the  most  promise,  assuming  that  the  BMDS  M&S  de¬ 
velopment  process  is  responsive,  timely,  and  credible. 
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BMDS  BDS  KINETIC  ENERGY  WEAPON  SYSTEM  REPRESENTATIONS 

KINECTIC  ENERGY 

CAPS 

MDWAR 

EADSIM 

EADTB 

MDSE 

SEA-BASE  KINETIC 

N 

N 

N 

N 

N 

SPACE-BASE  KINETIC 

N 

N 

N 

N 

N 

#  Avail  /Total/  %  Avail 

0  /  2  /  0% 

0  /  2  /  0% 

0  /  2  /  0% 

0  /  2  /  0% 

0  /  2  /  0% 

[SpaOO, 

[CM99, 

[BAC+95, 

[BBG+96, 

[Too94, 

SpaOl, 

TOM99, 

TOM97, 

RayOla, 

McQ97, 

Citation  from 

BBG+99f] 

TRWOOa, 

BBG+99e, 

Rayo2a, 

TEM00] 

TRWOOb, 

TBE02a, 

RayOlb] 

TRW02b] 

TBE02b] 

Table  9-5.  BMDS  Boost  Defense  Segment  Kinetic  Energy  Representations 


4.  BMD  System-Level  M&S  Weapon  System  Representation  Summary 

Currently,  there  are  fourteen  U.S.  BMD  weapon  systems  in  four  separate  categories 
(e.g.,  six  TDS  missile  systems,  four  MDS  missile  systems,  two  directed  energy  systems, 
and  two  kinetic  energy  systems)  and  two  international  weapon  systems  modeled  across  the 
five  BMD  System-Level  M&S.  Projecting  future  requirements  for  US-only  weapon  sys¬ 
tem  representations  in  the  BMDS  weapon  system  category,  and  assuming  the  current  set  of 
weapon  systems,  there  is  a  potential  requirement  for  seventy  weapon  representations  in 
every  block  build  (e.g.,  Block  2004,  Block  2006. .  .Block  2016).  Assuming  a  requirement 
by  the  BMDS  system  engineering  stakeholders  for  future  block  system  representations,  and 
the  need  to  retain  configuration-managed  copies  of  past  and  present  system  representation 
configurations,  the  potential  population  for  credible  BMDS  weapon  system  representations 
may  soon  number  in  the  hundreds.  This  suggests  the  need  for  a  methodology  to  manage 
model  representation  variability  with  a  disciplined  configuration  management  process. 

Possible  future  requirements  for  additional  acceptable  BMDS  representations  are 
daunting  when  a  review  of  the  Block  2004  representation  inventory  reflects  only  thirty-six 
acceptable  TDS,  MDS,  and  BDS  weapon  system  representations,  or  fifty-one  percent  of 
acceptable  system  representations  available.  The  international  weapon  system  programs, 
Arrow  and  MEADS,  have  sixty  percent  (6  for  9)  acceptable  system  representations.  These 
statistics  do  not  indicate  any  level  of  weapon  representation  certification  or  validation  by 
the  respective  program  office. 
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5. 


BMD  System-Level  M&S  Sensor  System  Representations 


The  BMDS  terminal  defense  segment  radar  sensors  (Table  9-6)  have  the  highest 
percentage  of  system  representations  in  the  System-Level  M&S  with  a  one  hundred  percent 
fill  rate.  The  rationale  for  this  high  rate  is  the  same  reason  provided  earlier  in  this  section 
for  the  TDS  weapon  systems.  However,  as  with  the  missile  systems  weapon  category  there 
is  a  significant  decrease  in  system  representations  in  the  MDS  (Table  9-8)  and  BDS  (Table 
9-9)  segments,  with  the  largest  void  in  three  simulations,  EADSIM,  EADTB,  and  MDSE. 


BMDS  TDS  RADAR  SENSOR  SYSTEM  REPRESENTATIONS 

TDS  RADARS 

CAPS 

MDWAR 

EADSIM 

EADTB 

MDSE 

PAC-2  AN/MPQ-53 

Y 

Y 

Y 

Y 

Y 

PAC-3  AN/MPQ-65 

Y 

Y 

Y 

Y 

Y 

TPS-59  (USMC) 

Y 

Y 

Y 

Y 

Y 

TPS-75  (USAF) 

Y 

Y 

Y 

Y 

Y 

AEGIS  SPY- IB/D 

Y 

Y 

Y 

Y 

Y 

#  Avail  /Total/  %  Avail 

6/6  / 100% 

6  /  6  / 100% 

6  /  6  / 100% 

6  /  6  / 100% 

6  /  6  / 100% 

[SpaOO,  SpaOl, 

[CM99, 

[BAC+95, 

[BBG+96, 

[Too94, 

BBG+99f] 

TOM99, 

TOM97, 

RayOla, 

McQ97, 

Citation  from 

TRWOOa, 

BBG+99e, 

Rayo2a, 

TEM00] 

TRWOOb, 

TBE02a, 

RayOlb] 

TRW02b] 

TBE02b] 

Table  9-6.  BMDS  TDS  Radar  Sensor  System  Representations 


The  availability  of  CAPS  and  MDWAR  sensor  representations  for  the  International 
TDS  radars  in  Table  9-7  reflect  the  relatively  low  fidelity  of  radar  system  representations  in 
CAPS  and  MDWAR.  The  ARROW  sensor  representations  in  EADTB  and  MDSE  support 
international  interoperability  exercises  and  testing.  Although  the  current  fill  rate  for  Inter¬ 
national  TDS  radar  representations  is  sixty  percent  overall,  the  international  M&S  program 
is  expected  to  grow  significantly  requiring  the  rapid  addition  of  system  representations  of 
all  types. 


BMDS  INTERNATIONAL  TDS  RADAR  SENSOR  SYSTEM  REPRESENTATIONS 

INTL TDS  RADARS 

CAPS 

MDWAR 

EADSIM 

EADTB 

MDSE 

ARROW  RADAR 

Y 

Y 

Y 

Y 

Y 

MEADS  FIRE  CNTRL 

Y 

Y 

N 

N 

N 
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MEADS  SURVLNCE 

Y 

Y 

N 

N 

N 

#  Avail  /Total/  %  Avail 

3  /  3  / 100% 

3  /  3  / 100% 

1/3/33% 

1  /  3  /  33% 

1/3/33% 

[SpaOO,  SpaOl, 

[CM99, 

[BAC+95, 

[BBG+96, 

[Too94, 

BBG+99f[ 

TOM99, 

TOM97, 

RayOla, 

McQ97, 

Citation  from 

TRWOOa, 

BBG+99e, 

Rayo2a, 

TEM00] 

TRWOOb, 

TBE02a, 

RayOlb] 

TRW02b] 

TBE02b] 

Table  9-7.  BMDS  International  TDS  Sensor  System  Representations 


The  next  three  BMDS  sensor  system  representation  tables  represent  radar,  satellite  / 
infrared,  and  ABL-specific  sensor  systems.  Table  9-8  through  Table  9-9  develop  a  similar 
pattern,  high  availability  of  representations  at  the  lower-fidelity  spectrum  and  few  represen¬ 
tations  available  for  the  higher-fidelity  simulations,  EADTB  and  MDSE.  Table  9-8,  MDS 
radar  sensor  representations  achieve  only  a  fifty  percent  fill  rate  (15/30). 


BMDS  MDS  RADAR  SENSOR  SYSTEM  REPRESENTATIONS 

MDS  RADARS 

CAPS 

MDWAR 

EADSIM 

EADTB 

MDSE 

THAAD  PDRR 

Y 

Y 

Y 

Y 

Y 

THAAD EMD 

Y 

Y 

Y 

N 

N 

COBRA  DANE 

Y 

Y 

N 

N 

N 

UEWR 

Y 

Y 

N 

N 

N 

SEA-BASE  XBR 

Y 

N 

N 

N 

N 

GBR/XBR 

Y 

Y 

N 

N 

N 

#  Avail  /Total/  %  Avail 

6  /  6  / 100% 

5/6/83% 

2  /  6  /  33% 

1  /  6  / 17% 

1  /  6  / 17% 

[SpaOO,  SpaOl, 

[CM99, 

[BAC+95, 

[BBG+96, 

[Too94, 

BBG+99f] 

TOM99, 

TOM97, 

RayOla, 

McQ97, 

Citation  from 

TRWOOa, 

BBG+99e, 

Rayo2a, 

TEM00] 

TRWOOb, 

TBE02a, 

RayOlb] 

TRW02b] 

TBE02b] 

Table  9-8.  BMDS  MDS  Radar  Sensor  System  Representations 


Table  9-9,  BMDS  satellite  /  infrared  sensors,  account  for  only  forty  five  percent  (9/20)  of 
the  possible  sensor  representations. 
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BMDS  BDS  SATELLITE  /  IR  SENSOR  SYSTEM  REPRESENTATIONS 

SATELLITES  /  IR 

CAPS 

MDWAR 

EADSIM 

EADTB 

MDSE 

DSP  SATELLITES 

Y 

Y 

Y 

Y 

Y 

SBIRS-HIGH 

Y 

Y250 

N 

N 

N 

SSTS  (SBIRS-LOW) 

Y 

Y 

N 

N 

N 

RAMOS  (US/RF  Program) 

N 

N 

N 

N 

N 

#  Avail  /Total/  %  Avail 

3  /  4  /  75% 

3  /  4  /  75% 

1  /  4  /  25% 

1  /  4  /  25% 

1  /  4  /  25% 

[SpaOO,  SpaOl, 

[CM99, 

[BAC+95, 

[BBG+96, 

[Too94, 

BBG+99f| 

TOM99, 

TOM97, 

RayOla, 

McQ97, 

Citation  from 

TRWOOa, 

BBG+99e, 

Rayo2a, 

TEM00] 

TRWOOb, 

TBE02a, 

RayOlb] 

TRW02b| 

TBE02b] 

Table  9-9.  BMDS  BDS  Satellite  /  Infrared  (IR)  Sensor  System  Representations 


Table  9-10  lists  the  Airborne  Laser  (ABL)  sensor  representations,  which  attain  a  sixty  per¬ 
cent  availability  rate  (9/15)  of  ABL  representations,  assisted  by  embedded  model  and  fed¬ 
eration  techniques. 


BMDS  ABL  SENSOR  SYSTEM  REPRESENTATIONS 

ABL  SENSOR 

CAPS 

MDWAR 

EADSIM 

EADTB 

MDSE 

BEACON  ILLUM. 

Y 

Y 

Y 

N 

N 

TRACKING  ILLUM. 

Y 

Y 

Y 

N 

N 

LASER  RANGER 

Y 

Y 

Y 

N 

N 

#  Avail  /Total/  %  Avail 

3  /  3  / 100% 

3  /  3  / 100% 

3/3/100% 

0  /  3  /  0% 

0  /  3  /  0% 

[SpaOO,  SpaOl, 

[CM99, 

[BAC+95, 

[BBG+96, 

[Too94, 

BBG+99f] 

TOM99, 

TOM97, 

RayOla, 

McQ97, 

Citation  from 

TRWOOa, 

BBG+99e, 

Rayo2a, 

TEM00] 

TRWOOb, 

TBE02a, 

RayOlb] 

TRW02b| 

TBE02b] 

Table  9-10.  BMDS  Airborne  Laser  Sensor  System  Representations 


6.  BMD  System-Level  M&S  Sensor  System  Representations  Summary 

There  are  eighteen  BMDS  sensor  systems  modeled  across  the  five  BMD  System- 
Level  simulations.  Out  of  a  possible  ninety  sensor  system  representations,  fifty-eight  are 
available,  a  sixty  four  percent  availability  rate.  Individual  sensor  category  statistics  range 
from  a  high  of  one  hundred  percent  for  the  five  sensor  system  representations  in  the  TDS 


SBIRS-High  and  SSTS/SBIRS-Low  are  represented  in  MDWAR  by  a  federation  with  the  Missile  Defense  Space  Tool  (MDST) 
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radar  sensor  category  to  a  low  of  forty-five  percent  in  the  satellite  /  infrared  sensor  cate¬ 
gory.  The  international  sensor  system  category  had  representations  for  sixty  percent  of  the 
simulations  under  study. 

7.  BMD  System-Level  M&S  Battle  Management  System  Representations 

The  BMDS  battle  management  (BM)  software  is  critical  to  the  success  of  missile 
defense,  a  fact  recognized  from  the  earliest  SDI  studies  including  the  Fletcher  Report 
[FMA+84]  and  Eastport  Study  Group  [CCL+85],  and  its  success  depends  largely  on  M&S 
support.  Command  and  control  (C2)  is  the  human  element  of  the  BM/C2,  supported  by  the 
automated  BM  functions  and  communications  network.  The  BMDS  Battle  Management 
system  representation  will  address  the  system  wide  BMDS  BM  capabilities  including  the 
engineering  needed  to  resolve  protocol  differences  between  TADIL-J  and  TADIL-K  mes¬ 
sage  formats  employed  by  the  TDS  and  MDS  segments  respectively,  integrate  the  separate 
battle  managers  developed  by  existing  MDA  programs,  and  support  BMDS  system-level 
future  battle  manager  development. 

BMDS  BM  systems  cited  in  Table  9-11  represent  high  risks  for  the  BMDS.  The 

Fletcher  Report  [FMA+84]  provided  strong  justification  for  reliable,  safe,  effective  missile 

defense  software  and  concluded: 

Further  emphasis  is  needed  on  simulation  as  a  means  to  assist 
the  design  of  battle  management  systems  and  software.  Spe¬ 
cific  work  is  needed  on  algorithms  related  to  critical  battle 
management  functions  [FMA+84]. 

Currently,  the  development  of  the  BMDS  System-Level  BM  is  in  the  embryonic 
stage  and  not  yet  adequately  represented  in  any  of  the  BMDS  System-Level  M&S.  How¬ 
ever,  BMD  element-level  legacy  BM  systems  exist  and  many  of  their  representations  exist 
with  the  BMD  System-levels  simulations.  The  five  TDS  battle  managers  allocated  to  the 
five  BMD  System-level  simulations  noted  in  Table  9-11  have  twenty- five  acceptable  model 
representations  out  of  a  possible  twenty-five  instantiations  for  a  one  hundred  percent  avail¬ 
ability  rate.  Acceptable  GMD  BM  representations  reside  in  both  CAPS  and  MDWARS, 
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although  missing  BM  representations  in  EADSIM,  EADTB,  and  CAPS,  create  an  overall 
forty  percent  BM  system  availability  rate. 

The  two  BDS  battle  managers,  ABL  and  SBL,  available  in  only  two  of  ten  possible 
BM  representations,  reflect  a  twenty  percent  availability  rate.  Table  9-11  lists  the  available 
representations  and  percent  availability  for  each  simulation.  The  entire  BM  representation 
population  in  Table  9-11  has  an  overall  sixty-four  percent  (29/45)  availability. 


BMDS  BATTL  EMANAGER  SYSTEM  REPRESENTATIONS 

C2BM 

CAPS 

MDWAR 

EADSIM 

EADTB 

MDSE 

BMDS  C2BM  -  System-Level 

N 

N 

N 

N 

N 

PAC-2  BMC2  -  TDS 

Y 

Y 

Y 

Y 

Y 

PAC-3  BMC2-PDB  5+  -  TDS 

Y 

Y 

Y 

Y 

Y 

AEGIS  BASELINE  5  -  TDS 

Y 

Y 

Y 

Y 

Y 

.IT AGS/ ALERT  -  TDS 

Y 

Y 

Y 

Y 

Y 

THAAD  BMC3  -  TDS 

Y 

Y 

Y 

Y 

Y 

GMD  BMC2  -  MDS 

Y 

Y 

N 

N 

N 

ABL  BMC41  -  BDS 

Y 

Y 

N 

N 

N 

SBL  BMC2  -  BDS 

N 

N 

N 

N 

N 

#  Avail  /Total/  %  Avail 

7/9  /  78% 

7  /  9  /  78% 

5/9  /  56% 

5  /  9  /  56% 

5/9  /  56% 

[SpaOO,  SpaOl, 

[CM99, 

[BAC+95, 

[BBG+96, 

[Too94, 

BBG+99f] 

TOM99, 

TOM97, 

RayOla, 

McQ97, 

Citation  from 

TRWOOa, 

BBG+99e, 

Rayo2a, 

TEMOO] 

TRWOOb, 

TBE02a, 

RayOlb] 

TRW02b| 

TBE02b] 

Table  9-11.  BMDS  Battle  Manager  System  Representations 


The  International  BM  system  representations  for  the  ARROW  and  MEADS  representations 
in  Table  9-12,  achieved  a  slightly  lower  fifty  percent  fill  rate,  with  ARROW  BM  repre¬ 
sented  at  a  one  hundred  percent  fill. 


BMDS  INTERNATIONAL  BM  SYSTEM  REPRESENTATIONS 

INTL  C2BM 

CAPS 

MDWAR 

EADSIM 

EADTB 

MDSE 

ARROW  BMC2 

Y 

Y 

Y 

Y 

Y251 

MEADS  BMC2 

N 

N 

N 

N 

N 

#  Avail  /Total/  %  Avail 

1  /  2  /  50% 

1  /  2  /  50% 

1  /  2  /  50% 

1  /  2  /  50% 

1  /  2  /  50% 

The  ARROW-MDSE  interoperability  program  achieves  this  capability. 

242  * 


Citation  from 

[SpaOO,  SpaOl, 

BBG+99f] 

[CM99, 

TOM99, 

TRWOOa, 

TRWOOb, 

TRW02b| 

[BAC+95, 

TOM97, 

BBG+99e, 

TBE02a, 

TBE02b] 

[BBG+96, 

RayOla, 

Rayo2a, 

RayOlb] 

[Too94, 

McQ97, 

TEMOO] 

Table  9-12. 

BMDS  International  B 

M  System  Representations 

8.  BMD  System-Level  M&S  Communication  System  Representations 

The  BMD  system-Level  M&S  communication  network  representations  in  Table  9- 
13  include  the  Joint  Data  Network  (JDN),  Joint  Planning  Network  (JPN),  and  the  Joint 
Composite  Tracking  Network  (JCTN).  The  BMD  System-Level  M&S,  except  CAPS,  ex¬ 
plicitly  model  the  JDN,  which  forms  the  communication  network  for  the  TDS.  The  ab¬ 
sence  of  explicit  communication  representations  in  CAPS  does  not  degrade  the  simulation, 
since  it  is  not  a  communications  planner.  EADTB  has  the  highest  percent  fill  of  communi¬ 
cation  representations,  confirming  its  primary  role  as  a  communications  interoperability 
tool.  Overall  the  three  communication  networks  have  five  of  fifteen  possible  representa¬ 
tions  in  the  BMD  System-Level  M&S,  an  availability  rate  of  thirty-three  percent. 


BMDS  COMMUNICATION  SYSTEM  REPRESENTATIONS 

COMMUNICATIONS 

CAPS252 

MDWAR 

EADSIM 

EADTB 

MDSE 

JDN253 

N 

Y 

Y 

Y 

Y 

JPN254 

N 

N 

N 

N 

N 

JCTN255 

N 

N 

N 

Y 

N 

#  Avail  /Total/  %  Avail 

0  /  3  /  0% 

1  /  3  /  33% 

1/3/33% 

2  /  3  /  66% 

1/3/33% 

[SpaOO,  SpaOl, 

[CM99, 

[BAC+95, 

[BBG+96, 

[Too94, 

BBG+99f[ 

TOM99, 

TOM97, 

RayOla, 

McQ97, 

Citation  from 

TRWOOa, 

BBG+99e, 

Rayo2a, 

TEMOO] 

TRWOOb, 

TBE02a, 

RayOlb] 

TRW02b] 

TBE02b] 

Table  9-13.  BMDS  Communication  System  Representations 


252 

CAPS  network  are  generic  with  no  explicitly  documented  network  models,  although  the  JDN  is  the  implicit  model  [BBG+99f|. 

253 

The  Joint  Data  Network  (JDN)  is  comprised  of  DoD  Link  11,  16,  and  22  message  protocols  sharing  sensor  data  between  all  subscrib¬ 
ers  in  near-real-time. 

254 

The  Joint  Planning  Network  (  JPN)  supports  the  broadcast  of  TRAP  (TDS)  and  TIBS  traffic  and  non-real-time  status  and  planning. 

255 

The  Joint  Composite  Tracking  Network  (JCTN)  provides  for  the  distribution  of  best  quality  engagement  data  to  all  subscribers,  in  as 
close  to  real-time  as  possible,  and  supports  the  extension  of  the  Navy’s  Cooperative  Engagement  Capability  (CEC). 
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9. 


BMD  System-Level  M&S  Platform  System  Representations 


The  BMDS  System-Level  Model  and  Simulation  Platfonn  Representations  data  in 
Table  9-14  shows  a  sixty- four  percent  availability  rate  of  launcher  platfonns,  with  the 
highest  density  of  available  representations  in  the  TDS  systems  designed  for  theater  mobil¬ 
ity  (e.g.,  PAC-2,  PAC-3,  THAAD)  and  the  Aegis  Weapon  System.  The  CAPS  model,  as 
noted  with  the  communication  systems,  operates  at  a  high  level  of  aggregation  and  does  not 
require  detailed  platform  information.  The  ABL  aircraft,  a  YAL-IA  747-400,  requires  the 
development  of  a  specific  track  configuration  in  all  the  System-Level  M&S  in  support  of 
its  current  doctrinal  employment  strategy. 


BMDS  PLATFORM  SYSTEM  REPRESENTATIONS 

PLATFORMS 

CAPS 

MDWAR 

EADSIM 

EADTB 

MDSE 

ABL  YAL-IA  747-400 

N 

N 

N 

N 

N 

THAAD  LAUNCHER 

N 

Y 

Y 

Y 

Y 

PAC-2  LAUNCHER 

N 

Y 

Y 

Y 

Y 

PAC-3  LAUNCHER 

N 

Y 

Y 

Y 

Y 

AEGIS  WPN  SYS 

N 

Y 

Y 

Y 

Y 

#  Avail  /Total/  %  Avail 

0  /  5  /  0% 

4  /  5  /  80% 

4  /  5  /  80% 

4  /  5  /  80% 

4  /  5  /  80% 

[SpaOO,  SpaOl, 

[CM99, 

[BAC+95, 

[BBG+96, 

[Too94, 

BBG+99f] 

TOM99, 

TOM97, 

RayOla, 

McQ97, 

Citation  from 

TRWOOa, 

BBG+99e, 

Rayo2a, 

TEM00] 

TRWOOb, 

TBE02a, 

RayOlb] 

TRW02b| 

TBE02b] 

Table  9-14.  BMDS  Platfonn  System  Representations 


10.  BMD  System-Level  M&S  Geodetic  Coordinate  Representations 

Geospatial  infonnation  provides  the  capability  to  describe  a  location  or  position, 
nonnally  exchanged  through  the  use  of  coordinate  locations.  Common,  well-known  geo¬ 
spatial  reference  systems  support  interoperability  with  accurate  coordinate  locations  and 
support  the  transformation  of  a  coordinate  system  into  multiple  definitions  of  geo-  and  non¬ 
geo-referenced  space.  The  coordinate  systems  include: 

•  Inertial  coordinates, 

•  Earth-centered,  Earth-Fixed  coordinates, 

•  Local  coordinates  [WilOO], 
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The  inertial  coordinate  systems  (e.g.,  non-rotating  coordinates)  have  an  orientation 
fixed  in  space,  with  the  Earth-Centered  Inertial  Model  (ECI)  coordinate  frame  commonly 
employed  to  define  the  motions  of  ballistic  missiles  or  spacecraft  [WilOO].  Spatial  informa¬ 
tion  consists  of  wide  population  of  measurements  and  data  types.  Earth-centric  spatial  lo¬ 
cations  include  an  Earth-centric  view,  indicating  that  spatial  locations  are  geospatial.  Spa¬ 
tial  locations  also  support  the  identification  of  a  fixed  reference  frame  rotating  with  the 
earth,  the  geographic  or  geodetic  reference.  Sensors  may  use  local  coordinate  systems  to 
measure  the  positions  of  other  bodies  relative  to  the  position  of  the  sensor(s)  [WilOO], 
Conversely  there  is  also  a  significant  amount  of  spatial  data  (e.g.,  astronomical,  orbital, 
geomagnetic  and  local  observations)  with  reference  frameworks  fixed  on  the  observer,  so¬ 
lar,  celestial  or  other  positional  standards.  Geospatial  information  interoperability  requires 
that: 

•  Coordinate  systems  be  defined  such  that  coordinates  describe  location  uniquely, 

•  A  data  transfer  mechanism  exists  to  transfer  data  to  other  defined  coordinate  sys¬ 
tems  [Bir97]. 

The  approximate  shape  of  the  Earth,  originally  defined  by  early  navigators  as  a 
spherical  body  supported  the  development  of  geodetic  coordinate  (GDC)  systems.  How¬ 
ever,  more  accurate  geodetic  coordinates  requires  the  modeling  of  Earth  as  an  oblate  sphe¬ 
roid  or  ellipsoid-of-rotation  [Bir97].  GDCs,  composed  of  the  ellipsoid  and  the  datum,  re¬ 
late  Earth-centered,  angular,  geodetic  latitude256  and  longitude257  to  an  actual  point,  near  or 
on  the  Earth’s  surface.  The  ellipsoid  is  a  good  approximation  of  the  geoid"  and  used  for 
many  mapping  applications.  Many  ellipsoids  model  the  earth,  and  each  ellipsoid  may  be 
defined  in  many  ways  in  reference  to  the  Earth.  The  datum  defines  the  location  of  the  el¬ 
lipsoid  with  respect  to  the  Earth. 

Modem  GDCs,  derived  from  satellite  data,  provide  measurements  defining  Earth- 
based  ellipsoids  with  accurate  global  data.  The  World  Geodetic  System  1984  [WGS84] 


~56  Latitude  is  the  angle  subtended  with  the  ellipsoid’s  equatorial  plane  by  a  perpendicular  through  the  surface  of  the  ellipsoid  from  a 
point,  with  positive  latitude  north  of  the  equator  and  negative  latitude  south  of  the  equator  [Bir97] 

257 

Longitude  is  defined  as  the  angle  measured  about  the  minor,  or  polar  axis  of  the  ellipsoid  from  a  prime  meridian  to  the  meridian 
through  a  point,  positive  to  the  east  of  the  prime  meridian  (e.g.,  Greenwich,  England)  or  negative  if  west  of  the  prime  meridian  [Bir97], 
~58  The  geoid  is  a  physical  surface  of  equi-gravitational  potential  corresponding  to  mean  sea-level,  extending  into  landforms  as  an  ap¬ 
proximation  of  mean  sea-level,  providing  surface  from  which  elevations  may  be  directly  measured  [Bir97] 
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developed  by  the  National  Imagery  and  Mapping  Agency  (formerly  the  Defense  Mapping 
Agency),  defines  the  current  Department  standard  datum  and  geoid,  and  employs  a  consis¬ 
tent  global  set  of  3 -dimensional  station  coordinates  inferring  the  location  of  an  origin,  the 
orientation  of  an  orthogonal  set  of  Cartesian  axes,  and  a  scale.  The  [WGS84]  standard, 
current  as  of  1996,  uses  the  refined  coordinates  of  the  pennanent  Department’s  Global  Po¬ 
sitioning  System  (GPS)  monitor  stations  as  the  operational  WGS-84  reference  frame. 


BMDS  GEODETIC  COORDINATE  REPRESENTATIONS 

GEODETIC  COORDINATE 

CAPS 

MDWAR 

EADSIM 

EADTB 

MDSE 

WGS-84 

N 

Y 

Y 

N 

Y 

Round,  Rotating  Earth 

Y 

N 

N 

Y 

N 

Non-rotating  Earth 

Y 

N 

N 

N 

N 

Earth  Centered  Inertial  (ECI) 

N 

Y 

N 

N 

N 

Locally  Spherical  Earth 

N 

N 

N 

N 

N 

#  Avail  /Total/  %  Avail 

2  /  5  /  40% 

2  /  5  /  40% 

1  /  5  /  20% 

1  /  5  /  20% 

1  /  5  /  20% 

[SpaOO,  SpaOl, 

[CM99, 

[BAC+95, 

[BBG+96, 

[Too94, 

BBG+99f] 

TOM99, 

TOM97, 

RayOla, 

McQ97, 

Citation  from 

TRWOOa, 

BBG+99e, 

Rayo2a, 

TEM00] 

TRWOOb, 

TBE02a, 

RayOlb] 

TRW02b| 

TBE02b] 

Table  9-15.  BMDS  Geodetic  Coordinate  Representations 


In  addition,  a  new  global  model  of  the  Earth’s  gravitational  field,  the  Earth  Gravita¬ 
tional  Model  1996  (EGM-96),  replaced  the  outdated  WGS-84  gravitational  model 
[WGS84],  WGS-84  is  the  current  JTA  geospatial  data  interchange  standard  and  prescribed 
with  the  EGM-96  for  the  BMDS  positioning,  navigation,  and  timing  standards  [WBF+00]. 
The  EGM-96  supports  the  accurate  assessment  of  the  Earth’s  gravitational  function  sup¬ 
porting  the  prediction  of  an  object’s  state  vector  in  time,  a  critical  component  and  a  possi¬ 
ble  source  of  heterogeneous  system  representation  anomalies  in  threat  and  BMDS  compo¬ 
nent  trajectories,  sensor  target  tracking,  geodetic  information  exchange  or  target  track  data 
(e.g.,  substantive  interoperability  anomaly)  if  missing  or  modeled  incorrectly  [WBF+00]. 
Table  9-15  lists  the  status  of  geodetic  coordinate  representations  in  the  BMD  System-level 
M&S.  Other  environmental  models  currently  under  assessment  for  standardization  and  in¬ 
clusion  in  the  set  of  BMD  System-level  M&S  include  local  weather  models,  earth  atmos¬ 
phere  models,  sun  position,  lunar  position,  and  star  catalog  [Wil02,  Wil03]. 
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11.  BMD  System-Level  M&S  Summary  of  Representations 


The  summary  of  acceptable  authoritative  representations  in  Table  9-16  of  the  five 
BMDS  simulations  under  study  provides  the  following  statistics: 

•  All  five  M&S  systems  included  representations  in  the  weapon,  sensor,  communica¬ 
tion,  platform,  and  BM/C2  categories  or  product  lines. 

•  CAPS  accounted  for  the  highest  percentage  of  authoritative  representations  in  the 
missile  and  sensor  product  lines  with  eighty-two  percent  and  ninety-five  percent, 
respectively;  and  tied  with  MDWAR  in  the  BM/C2  product  line  with  seventy-eight 
percent.  The  ninety-five  percent  level  for  the  CAPS  sensor  product  line  was  the 
highest  percentage  of  authoritative  representations  in  a  product  line  found  in  the 
study.  Conversely,  CAPS  also  had  the  only  two  product  line  categories,  the  com¬ 
munication  and  platfonn  with  no  authoritative  representations.  The  level  of  aggre¬ 
gation  and  abstraction  in  authoritative  representations  common  to  both  CAPS  and 
MDWAR  contributed  to  the  CAPS  and  MDWAR  statistics. 

•  The  weapon  product  line  authoritative  representations  ranged  from  a  high  of  eighty- 
two  percent  to  thirty-eight  percent.  The  mode  for  weapon  authoritative  representa¬ 
tions  was  thirty-eight  percent. 

•  The  sensor  product  line  authoritative  representations  ranged  from  a  high  of  ninety- 
five  percent  to  forty-one  percent.  The  mode  for  sensor  authoritative  representations 
was  forty-one  percent. 

•  The  BM/C2  product  line  authoritative  representations  ranged  from  a  high  of  sev¬ 
enty-eight  percent  to  fifty-six  percent.  The  mode  for  BM/C2  authoritative  represen¬ 
tations  was  fifty-six  percent. 

•  The  communications  product  line  authoritative  representations  ranged  from  a  high 
of  sixty-six  percent  to  zero.  The  mode  for  communications  authoritative  represen¬ 
tations  was  thirty-three  percent. 

•  The  platfonn  product  line  authoritative  representations  ranged  from  a  high  of  eighty 
percent  to  zero.  The  mode  for  platfonn  authoritative  representations  was  eighty 
percent,  the  highest  mode  statistic  in  the  study. 

•  The  five-simulation  total  product  line  authoritative  representations  ranged  from  a 
high  of  seventy-five  percent  to  forty-seven  percent.  The  mode  for  the  five  simula¬ 
tion  total  authoritative  representations  was  fort-seven  percent. 

•  The  five-simulation  population  including  all  product  line  authoritative  representa¬ 
tions  equaled  sixty  percent. 


BMDS  SYSTEM-LEVEL  PRODUCT  LINE  REPRESENTATION  SUMMARY 


SYSTEM  REPRESENTATIONS 

CAPS 

MDWAR 

EADSIM 

EADTB 

MDSE 

TDS  Missiles 

6  /  6  / 100% 

5  /  6  /  83% 

5  /  6  /  83% 

5  /  6  /  83% 

5/6/83% 

International  TDS  Missiles 

2/2/100% 

1  /  2  /  50% 

1  /  2  /  50% 

1  /  2  /  50% 

1  /  2  /  50% 

MDS  Missiles 

4  /  4  / 100% 

3  /  4  /  75% 

1  /  4  /  25% 

0  /  4  /  0% 

0  /  4  /  0% 

Directed  Energy  Weapons 

1  /  2  /  50% 

1  /  2  /  50% 

1  /  2  /  50% 

0/2 / 0% 

0  /  2  /  0% 
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Kinetic  Energy  Weapons 

0  /  2  /  0% 

0121 0% 

0  /  2  /  0% 

0  /  2  /  0% 

0121 0% 

WEAPON  SUB-TOTAL 

13  / 16  /  82% 

10  / 16  /  63% 

8  / 16  /  50% 

6  / 16  /  38% 

6  / 16  /  38% 

TDS  Radars 

6  /  6  / 100% 

6161 100% 

6161 100% 

6161 100% 

6161 100% 

International  TDS  Radars 

3/3/100% 

3  /  3  / 100% 

1/3/33% 

1  /  3  /  33% 

1/3/33% 

MDS  Radars 

6  /  6  / 100% 

5/6/83% 

2/6/33% 

1  /  6  / 17% 

1  /  6  / 17% 

Satellite  /  IR  Sensors 

3  /  4  /  75% 

3  /  4  /  75% 

1  /  4  /  25% 

1  /  4  /  25% 

1  /  4  /  25% 

ABL  Sensors 

3/3/100% 

3  /  3  / 100% 

3/3/100% 

0  /  3  /  0% 

0  /  3  /  0% 

SENSOR  SUB-TOTAL 

21/22/95% 

20/22  /91% 

13  1221  59% 

9/22/41% 

9/22/41% 

BMDS  C2BM 

7  /  9  / 78% 

119/  78% 

5/9  /  56% 

5  /  9  /  56% 

5/9  /  56% 

International  C2BM 

1  /  2  /  50% 

1  /  2  /  50% 

1  /  2  /  50% 

1  /  2  /  50% 

1  /  2  /  50% 

SENSOR  SUB-TOTAL 

8/11/73% 

8/11/73% 

6/11/55% 

6  / 11  /  55% 

6  / 11  /  55% 

COMM  SYSTEM  SUB-TOTAL 

0  /  3  /  0% 

1  /  3  /  33% 

1/3/33% 

2  /  3  /  66% 

1/3/33% 

PLATFORM  SUB  TOTAL 

0  /  5  /  0% 

4  /  5  /  80% 

4  /  5  /  80% 

4/5  /  80% 

4  /  5  /  80% 

SIMULATION  TOTAL 

42  /  57  /  74% 

43  /  57  /  75% 

32  /  57  /  56% 

21/511  47% 

26  /  57  /  47% 

POPULATION  TOTAL 

170  /  285  /  60% 

Table  9-16.  BMD  System-Level  M&S  Representation  Summaries 


Although  the  five  simulations  equate  to  a  statistically  insufficient  sample  to  relate  to 
the  Department’s  total  M&S  population,  the  results  correlate  with  the  numerous  studies, 
reports,  and  papers  cited  in  earlier  chapters,  suggesting  the  Department  population  repre¬ 
sentations  as  a  whole,  may  be  more  unpredictable  than  authoritative.  However,  this  study 
covers  one  hundred  percent  of  the  system-level  model  representation  population,  with  a 
significant  sample  size  of  285  representations,  which  viewed  in  the  Table  9-16,  forms  pat¬ 
terns.  [Fow97]  defines  a  pattern  as  “an  idea  that  has  been  useful  in  one  practical  and  will 
probably  be  useful  in  others”  [Fow97],  and  further  suggests  that  the  establishment  of  a 
common  framework  supports  the  concept  of  component  reuse  for  information  systems. 

Table  9-16  identifies  potential  patterns  for  several  software  architecture  patterns  in¬ 
cluding  product  families  (e.g.  weapons),  software  product  lines  (e.g.  missiles,  lasers),  soft¬ 
ware  product  line  systems  (e.g.  PAC-3,  THAAD  missiles),  and  a  software  product  line  ar¬ 
chitecture.  The  sixty  percent  availability  rate  of  authoritative  representations  (e.g.,  model 
product  lines)  resulted  from  the  causal  reasons  cited  in  earlier  chapters  and  for  the  specific 
reasons  noted  in  this  chapter.  This  suggests  that  current  Department  simulation  software 
development  techniques,  procedures,  and  practices  are  inadequate  or  incomplete,  inconsis¬ 
tently  implemented,  or  a  combination  of  both  factors. 


248 


The  BMDS  poses  even  more  challenges  to  the  current  status.  The  BMDS  program 
requires  authoritative  representations  for  simulations  supporting  the  engineering  of  future 
BMDS  block  capabilities.  The  BMDS  system  engineering  process  requires  authoritative 
representations  for  Blocks  2004,  2006,  2008,  2010,  2012,  2014  and  beyond  to  explore  and 
experiment  with  capabilities,  and  provide  a  means  to  extend  the  perfonnance  window  of 
existing  system.  Due  to  live-test  limitations,  the  BMDS  System-Level  M&S  will  provide  a 
significant  percentage  of  the  future  data  needed  to  assess  BMD  System-wide  interoperabil¬ 
ity  and  performance.  The  urgency  of  improving  the  BMDS  System-Level  M&S  and  add¬ 
ing  more  credible  authoritative  representations  increased  dramatically  with  the  President’s 
decision  to  deploy  a  limited  Missile  Defense  capability  in  2004.  At  that  time  the  BMDS 
System-Level  M&S  will  support  an  operational  Missile  Defense  for  the  Nation,  allies,  and 
friends.  Chapter  X  provides  a  software  architecture-based  alternative  to  improve  the  level 
of  authoritative  representations  in  future  BMDS  System-Level  M&S. 

F.  BMDS  INTERNATIONAL  COOPERATIVE  M&S  PROGRAM 

The  Agency’s  international  cooperative  M&S  program  requires  the  development  of 
system  representations  with  the  necessary  detail,  attributes  [MDW+01],  and  goodness-to-fit 
must  pass  several  hurdles  from  the  formal  approval  of  the  international  agreement  permit¬ 
ting  the  initiative,  completion  of  required  disclosure  statement,  implementing  the  necessary 
contracting  vehicles  supporting  the  effort,  and  executing  the  actual  model  development  ef¬ 
fort.  The  international  cooperative  M&S  development  effort  must  also  address  infonnation 
assurance,  language,  cultural,  and  national  priority  issues,  in  addition  to  the  normal  cost, 
performance,  and  schedule  project  management  concerns.  All  or  any  one  of  these  condi¬ 
tions  may  provide  the  underlying  conditions  for  an  invalid  federate  or  the  creation  of  intol¬ 
erable  representational  anomalies,  resulting  in  substantive  interoperability  issues. 

Table  9-17  illustrates  the  scope  of  international  cooperative  M&S  initiatives  sup¬ 
ported  by  the  BMD  System-Level  M&S.  This  international  M&S  effort  is  a  key  compo¬ 
nent  of  the  missile  defense  interoperability  strategy  with  friends,  allies,  and  supports  evolv¬ 
ing  international  cooperation  efforts  with  new  partner  nation  initiatives  such  as  the  Russian 
Federation  Model  and  Simulation  Program  (US/RF).  Table  9-17  reflects  the  System-Level 
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M&S  supporting  the  initiatives,  and  the  type  of  international  agreements  authorizing  the 
program.  EADSIM  supports  every  international  cooperative  M&S  program,  except  the 
embryonic  US/RF  initiative,  while  NATO  employs  EADTB  in  missile  defense  interopera¬ 
bility  studies.  The  ARROW-MDSE  program  directly  supports  the  international  tenninal 
defense  segment  interoperability  testing  of  the  U.S.  PATRIOT  missile  defense  system  with 
the  Israeli  ARROW  missile  defense  system. 


COUNTRY 

SYSTEM-LEVEL  M&S 

LEGAL  BASIS 

CAPS 

EADSIM 

EADTB 

MDSE 

MOA 

MOU 

D 

F 

w 

s 

Australia 

X 

X 

Canada 

X 

X 

Germany 

X 

X 

X 

X 

Greece 

X 

X 

Israel 

X 

X 

X 

Japan 

X 

X 

S.  Korea 

X 

X 

NAMEADSMA 

X 

X 

X 

NC3A 

X 

X 

X 

UAE 

X 

X 

UK 

X 

X 

X 

X 

Singapore 

X 

X 

Spain 

X 

X 

Netherlands 

X 

X 

X 

Turkey 

X 

X 

X 

NATO  FS 

X 

X 

X 

X 

US/RF 

New  Model  and  Simulation  Development  Program  -  Legal  Agreement  Pending 

TOTAL 

2 

16 

5 

1 

6 

1 

2 

6 

1 

2 

Table  9-17.  BMD  International  Cooperative  M&S  Program 

G.  BMDS  ELEMENT-LEVEL  MODELS  &  SIMULATIONS  PROGRAM 


The  major  Element-Level  M&S,  listed  in  Appendix  H  directly  support  the  program 
requirements  of  MDA  element  program  managers.  They  are  also  components  of  the  joint 
BMD  System  M&S  Hierarchy,  developed  in  accordance  with  the  approved  element  Model 
and  Simulation  Support  Plan  (MSSP).  Element-Level  M&S  employ  BMD  System-TSEL 
M&S  in  the  development  of  their  respective  system,  and  support  the  development  of  accu¬ 
rate  element  representations  in  the  BMD  System-Level  M&S  [Kad02a], 
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H.  BMDS-LEVEL  THREAT,  ENVIRONMENT,  SIGNATURE  &  LETHALITY 

MODELS  &  SIMULATIONS 


The  Legacy  BMD  System  Threat,  Environment,  Signature,  and  Lethality  (TSEL) 
M&S  are  the  foundation  codes  of  the  missile  defense  domain.  These  common-and  general- 
use  codes  represent  the  highest  fidelity  simulations  in  the  missile  defense  M&S  domain  and 
directly  support  the  design,  development,  and  testing  of  all  missile  defense  element  sys¬ 
tems.  These  codes  require  the  highest  level  of  credibility.  In  addition,  the  expanded  battle 
space  of  the  new  BMD  system  adds  significant  new  requirements,  beyond  the  current  ca¬ 
pability  of  the  tools,  necessitating  major  upgrades  and  new  development  programs.  Ap¬ 
pendix  I  has  a  short  list  of  the  BMDS  Threat,  Environment,  Signature,  and  Lethality  M&S 
program. 

I.  BMDS  SYSTEM-LEVEL  M&S  VV&A  PROGRAM 

1.  AGENCY  M&S  VV&A  EFFORTS  (1986-2000) 

The  BMD  Core  M&S  program  inherited  a  legacy  of  challenges,  issues,  and  prob¬ 
lems  requiring  timely  solutions  to  allow  the  successful  evolution  of  credible  Agency  M&S 
supporting  confidence  in  the  simulation  results.  The  agency  supported  multiple  M&S  re¬ 
ports,  studies,  analyses,  and  reviews  to  establish  the  credibility  of  the  M&S  supporting  the 
unprecedented  technical  challenge  of  the  missile  defense  domain  including  :  [FMA+84, 
CCL+85,  Bre88,  SCC+98,  HSW+92,  EHH97,  KBM+97,  Li97,  SEL97,  SEH+97,  EHH+98, 
WAA+98,  ZHF98,  BBG+99a,  BBG+99b,  BBG+99c,  KGB+99,  Li99,  ISE99,  WAA+99, 
RTM+99a,  RTM+99b,  RTM+99c,  RTM+99d,  RTM+99e,  RTM+99f,  RTM+99g,  BDH00, 
LEH+00,  SBH+01,  WelOO,  BRC+01,  CoyOl,  EP01,  LWC+01,  LHC+02,  Mil02a, 

STG+02]. 

The  Agency’s  internally-  and  externally-generated  reviews  for  existing  simulation 
systems  identified  the  same  type  of  problems  noted  in  the  preceding  generation  of  air  de¬ 
fense  simulations  (e.g.,  Adage,  Carmonette,  COMO  III  [Che87]).  The  results  also  con- 
finned  the  lack  of  silver-bullet  solutions,  and  closely  paralleled  the  credibility  issues  and 


251 


systemic,  persistent  Department  M&S  problems  outlined  in  the  preceding  chapters  includ¬ 
ing: 

•  Lack  of  Technical  and  Substantive  interoperability 

•  Need  to  improve  V&V  process, 

•  Lack  of  documentation,  including  conceptual  models, 

•  Poor  or  nonexistent  configuration  management  of  models  and  data, 

•  Inadequate  understanding  by  users  and  decision-makers  of  model  assumptions, 

•  Limitations,  and  capabilities, 

•  Insufficient  model  development  practices, 

•  Need  for  improved  procedures  to  update  and  maintain  models, 

•  Need  to  improve  how  the  M&S  credibility  is  established, 

•  Need  to  improve  how  the  validity  of  the  model  is  determined, 

Simulations  need  hard  data  from  test  programs. 

The  study  effort  for  the  ballistic  missile  defense  M&S  research  phase  started  in  Sep¬ 
tember  2000  and  included  a  review  of  the  agency  documents  including  multiple  Agency 
M&S  briefings,  reviews,  reports,  staffing  actions,  test  results,  conversations,  and  assess¬ 
ments,  cited  above.  The  Agency’s  VV&A  issues  during  the  1990s  mirrored  the  Depart¬ 
mental  M&S  VV&A  issues.  Documented  V&V  of  the  BMDO  core  model  set  during  the 
1986-1999  timeframe  was  inconsistent  and  rare.  Two  specific  studies  representative  of 
other  Agency  M&S  studies  spanning  most  of  the  1990s  include  [HSW+92]  and 
[BBG+99a], 

In  1992  [HSW+92]  reported  on  the  SDIO’s  Brilliant  Pebbles1  program  effective¬ 
ness  and  the  extensive  use  of  simulations  in  support  of  system  design,  manufacturing  and 
deployment;  concluding  the  simulations  were  still  immature,  and  used  unproven  assump¬ 
tions,  and  noting: 

The  accuracy  of  Brilliant  Pebbles’  simulations  has  not  been 
validated  by  test  results  or  verified  by  fonnal  reviews  of  the 
simulations  themselves  [HSW+92], 

The  BMDO  and  Joint  Theater  Air  and  Missile  Defense  Organization  (JTAMDO) 

conducted  a  joint  a,  M&S  review  in  1999  [BBG+99a]  of  selected  BMDO  core  models.  The 

report  identified  model  validation  among  the  key  risk  areas  and  noted: 

There  is  no  BMDO-wide  process  for  assuring  adequate 
model  validity.  The  degree  of  V&V  performed  is  currently 
up  to  the  organization  funding  the  developer,  and  no  stan- 
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dards  are  followed.  There  is  no  organized  link  with  the  TE 
[test  and  evaluation]  community  for  obtaining  data  for  vali¬ 
dation  or  to  show  that  the  models  can/cannot  predict  or  ex¬ 
tend  testing.  There  has  only  been  a  minimal  effort  to  com¬ 
pare  model  results  on  the  same  scenario  (anchoring)  and 
inadequate  checkout  of  less  detailed  models  against  results 
from  the  more  detailed  models  [BBG+99a], 

The  majority  of  the  [BBG+99a]  issues  included  systemic  BMDO  organizational,  process, 

and  technical  issues.  Accordingly,  the  joint  BMDO/JTAMDO  M&S  review  [BBG+99a] 

recommended  an  on  going  V&V  process  citing: 

V&V  is  another  important  activity  not  actively  pursued  at 
this  time  in  a  centralized  manner,  resulting  in  uncertainties 
about  the  robustness  of  our  simulations  and  non-uniformity 
in  the  degree  of  V&V  performed.  There  should  be  a 
planned  central  effort  to  validate  our  models,  collect  valida¬ 
tion  data,  and  assess  the  developers’  processes  in  order  to 
reduce  program  risk.  A  part  of  this  program  should  be  to 
link  T&E  activities  in  an  active  effort  to  obtain  data  for 
model  validation  and  for  making  sure  the  simulations  can 
validly  be  used  to  extend  test  data  for  model  validation  and 
for  making  sure  the  simulations  can  validly  be  used  to  ex¬ 
tend  test  data.  Anchoring  simulations  results  is  another 
critical  activity,  which  would  improve  our  understanding  of 
the  models  and  of  their  differences  and  which  would  in¬ 
crease  our  confidence  level  or  motivate  corrections 
[BBG+99a], 

[MS02]  and  [Lev02]  address  the  two  key  issues  of  credibility  and  confidence  in 

missile  defense  simulations,  with  [Lev02]  noting: 

Weapon  system  confidence  is  being  able  to  predict  the  sys¬ 
tem’s  performance  to  within  a  quantified  uncertainty  or 
confidence  interval  [Lev02]. 

However,  the  current  suite  of  BMD  System-Level  M&S  experienced  various  inconsistent 
VV&A  approaches  as  the  BMD  System  M&S  V&V  Program259  matured  [HSW+92, 
BBG+99a], 

In  1999  [BBG+99F]  identified  that  CAPS  lacked  a  formal  V&V  process  since  first 


)  The  Agency  implemented  a  centrally  managed  verification  and  validation  program  of  the  core  M&S  in  FY2002  establishing  a  V&V 
process  to  improve  systemic  credibility  issues  and  improve  user  confidence  in  the  Agency  core  M&S  results. 
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released  in  1994,  and  noted,  “there  is  no  indication  of  the  validity  of  the  CAPS  engagement 
logic  to  represent  BMDO  systems”  [BBG+99f].  A  documented  CMMS/  FDMS  for  CAPS 
was  never  developed.  However,  CAPS  was  formally  accredited  for  specific  use  on  two 
occasions: 

•  1994-1996  for  the  Theater  Missile  Defense  Cost  and  Operational  Effectiveness 
Analysis  study,  accredited  by  both  the  Army  and  the  Navy, 

•  1997  -  Accreditation  as  the  interim  Joint  Defensive  Planner  (JDP)  by  the  BMDO 
[BBG+99a]. 

MDWAR,  the  newest  addition  to  the  core  model  suite  initiated  a  V&V  program 
early  in  the  development  cycle.  The  first  major  MDWAR  wargame,  C2SIM99,  met  joint- 
accreditation  requirements  by  the  NMD  program  office  and  USSPACECOM  for  execution 
in  November  1999.  Follow-on  wargame  efforts  included  the  Theater  Ballistic  Missile  De¬ 
fense  Operator-In-The-Loop  Phase  I  war  game  exercise  in  July  2000,  and  Phase  II  war 
game  exercise  in  September  2000,  with  both  events  meeting  joint-accreditation  require¬ 
ments  from  the  NMD  program  office  and  USSPACECOM.  The  developer  first  published 
conceptual  model  documentation  for  MDWAR  components  in  1999  [TRW02a,  TRW02b]. 

EADSIM  developed  a  wide  missile  defense  community  acceptance  throughout  the 
1990s,  although  [JAS97c]  noted  in  a  comprehensive  review  that  little  fonnal  V&V  was 
complete,  no  documented  V&V  results,  and  verification  limited  by  the  lack  of  a  detailed 
design  specification  against  which  to  check  the  software  code.  [JAS97c]  acknowledged 
that  some  V&V  work  may  be  in  progress  at  the  time  or  classified,  although  AFOTEC  re¬ 
ported  similar  findings  in  1994.  EADTB  credibility  according  to  [JAS97c]  depended  al¬ 
most  completely  on  face  validation  assessments  and  community  acceptance. 

The  first  major  study  employment  of  the  EADTB  simulation  was  the  BMDO  Cost 
and  Operational  Effectiveness  Analysis  (COEA)  study.  A  face  assessment  [BBB+96]  re¬ 
view  of  EADTB  for  the  COEA  study  was  completed  October  25,  1996  and  identified  96 
issues  and  34  product  quality  issues.  [BBB+96]  detailed  major  problems  including: 

•  The  lack  of  requirements  against  which  to  measure  the  capabilities, 

•  Excessive  run-time, 

•  A  high  learning  curve, 

•  Insufficient  testing  through  November  1995, 

•  Verification  was  limited  to  algorithms,  code  testing,  and  performance,  with  no  at- 
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tempt  at  design  verification. 

•  An  incomplete  draft  VV&A  Plan,  dated  March  14,  1994  had  not  been  placed  on 
contract. 

•  V&V  contactor  had  one  person  assigned. 

•  No  formal  conceptual  model  deliverable. 

•  Documentation  was  a  high  concern  with  key  documents  outdated  or  missing 
[BBB+96], 

[BBB+96]  also  reviewed  EADTB  V&V  procedures  in  the  areas  of  ballistic  missile 
flights  and  radar  sensors,  and  noted  the  existing  EADTB  V&V  method  compared  results 
with  other  models  such  as  EADSIM  or  COMO.  [BBB+96]  recommended  EADTB  for  an 
application  “wring  out”  but  not  for  the  more  critical  and  contentious  missile  defense  cam¬ 
paign-level  studies. 

The  EADTB  program  established  a  fonnal  V&V  effort  for  the  simulation’s  CMS 
components  in  1993  and  initiated  a  follow-on  V&V  program  for  the  system  specific  repre¬ 
sentations  (SSR)  in  1999.  The  reasons  for  the  six  year  difference  between  the  1993  initia¬ 
tion  of  the  EADTB  CMS  V&V  program  and  the  1999  EADTB  system  representations 
V&V  program  highlight  systemic  issues  identified  in  this  study  limiting  the  credibility  of 
all  Agency  System-Level  M&S  during  the  1990s: 

•  In  the  competition  for  resources  between  developing  new  system  requirements  with 
supporting  stakeholders,  or  support  for  the  potentially  high  recurring  costs  of  estab¬ 
lishing  and  maintaining  a  consistent  V&V  program,  the  delivery  of  new  capabilities 
generated  user-support,  whereas,  V&V  program  efforts  provided  no  new  capabili¬ 
ties  or  satisfied  customers,  and  could  be  contentious, 

•  The  inherent  complexity  of  V&V  confused  most  stakeholders  and  decision-makers, 
detracting  from  V&V  proponent’s  ability  to  compete  with  other  Agency  require¬ 
ments  for  resources, 

•  A  concern  by  the  MDAP  program  offices  that  their  system  could  be  misrepresented 
in  a  simulation,  or  the  representation  used  for  a  purposes  it  was  not  designed  or  in¬ 
tended,  such  as  system  perfonnance, 

•  The  time  and  resources  supporting  SSR  development  in  the  MDAP  system  engi¬ 
neering  correspondingly  reduced  the  MDAP’s  resources  for  it’s  primary  mission, 
with  little  perceived  return-on-investment, 

•  The  simulation  developers  had  limited  access  to  the  MDAP  program  offices  to  re¬ 
search,  design  and  develop  credible  system  representations  for  Agency-level  stud¬ 
ies,  analyses,  or  tests. 
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•  Discrete  CMMS/FDMS,  prescribed  by  [DoD94,  DoD96]  were  not  developed  for 
any  of  the  five  simulations  during  the  early  life-cycle  phases,  creating  at  a  minimum 
the  possibility  of  logical  inconsistency  in  the  VV&A  process, 

•  Any  efforts  to  determine  the  credibility  of  a  simulation  or  federation  for  intended 
use  was  necessarily  ad  hoc,  expensive,  and  short-lived  without  a  verified  and  vali¬ 
dated  conceptual  model  of  the  mission  space  upon  which  to  base  the  verification 
and  validation  process  for  the  FOM  or  software  implementation, 

•  An  Agency  trend  developed  in  the  1990s  to  accredit  a  simulation  without  underly¬ 
ing  conceptual  models  of  the  mission  space  or  V&V  plans, 

•  The  Department’s  accreditation  process  supported  multiple  accreditation  options 
and  it  was  acceptable  to  accredit  the  simulation  by  simply  using  it, 

•  Face  validation  or  a  review  by  subject  matter  experts  of  the  simulation  for  an  in¬ 
tended  use  became  the  de  facto  standard  for  establishing  credibility  and  supporting 
user  confidence, 

•  The  V&V  of  certain  SSRs  (e.g.,  threats,  systems  under  development)  was  extremely 
complicated  and  required  close  coordination  with  many  agencies, 

•  Contractors  had  no  incentive  to  V&V  simulations,  and  contractual  language  sup¬ 
porting  V&V  as  a  deliverable  were  few, 

•  The  previously  discussed  Year  2000  remediation  effort  and  HLA  compliancy  work 
pushed  V&V  efforts  further  down  in  the  requirement  queue. 

2.  AGENCY  M&S  VV&A  EFFORTS  (2001-2003) 

Two  BMD  System-level  M&S,  EADTB  and  MDSE,  supported  a  major-program 
testing  event,  the  PAC-3  independent  operational  test  and  evaluation  (IOT&E)  with  the 
PATRIOT  IOT&E  Interoperability  Demonstration  (PHD)  in  April  2002.  The  event  was 
significant  in  that  the  Army  Test  and  Evaluation  Command  accredited  the  Missile  Defense 
System  Exerciser  [Bar02a,  CRC02a],  in  a  virtual  hardware-in-the-loop  stimulation  tool 
role,  and  the  Extended  Air  Defense  Test  Bed  [Bar02b,  CRC02b,  KS02],  as  a  constructive 
simulation  for  a  full-up  five  firing  unit  air  defense  battalion  in  a  representative  threat  envi¬ 
ronment  supplementing  a  Department  IOT&E  for  the  PAC-3  missile  defense  system.  The 
PHD  built  on  a  highly  successful  PAC-3  Developmental  test  and  M&S  program  [BRC+01], 
The  year-long  preparation  effort  in  support  of  the  PAC-3  PHD  required  the  full  co¬ 
operation  of  the  Agency  M&S  organizations,  program  offices,  simulation  developers, 
VV&A  agents,  the  operational  test  community  and  many  other  stakeholders  to  develop, 
verify,  validate,  accredit  or  certify  the  system  representations  as  accurate  for  the  intended 
use  of  testing  PAC-3  interoperability.  The  interoperability  assessment  addressed  PAC-3 
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intra-battalion  interoperability,  interoperability  between  the  Patriot  system  and  other  ele¬ 
ments  of  the  Anny  Air  and  Missile  Defense  task  Force,  and  interoperability  between  Pa¬ 
triot  and  other  Joint  Service  elements  in  a  Joint  environment  scenario  [Bar02a,  Bar02b]. 

The  EADTB  and  MDSE  simulated  Theater  Missile  Defense  architecture  with  tactically 
representative  threats,  message  traffic,  and  communications  loading.  The  PHD  met  all  test 
objectives  and  successfully  completed  all  test  runs-for-record  ahead  of  schedule  in  April 
2002. 

Based  on  past  perfonnance  and  it’s  credibility  in  the  missile  defense  domain, 

MDSE  was  incorporated  into  the  Department’s  evolving  Joint  Distributed  Engineering 
Plant  (JDEP)  program  [DahOla,  DahOlb,  DTOla,  DTOlb,  Lew02b,  DC02,  Rad02]  support¬ 
ing  improved  interoperability  of  Joint  Forces,  an  example  of  large-scale  reuse  by  the  De¬ 
partment  and  a  potentially  significant  return-on-investment.  Even  more  importantly,  the 
collaborative  multi-stakeholder  effort  supporting  the  PHD  set  the  foundation  for  systemi- 
cally  developing  credible  system  representations  in  all  BMD  System-level  simulations  with 
element  program  participation  [Kad02a]. 

The  Extended  Air  Defense  Simulation  supported  the  Joint  Project  Optic  Windmill 
(JPOW)  VII  Exercise  [Obe02]  in  September  2002,  after  completing  a  collaborative  VV&A 
process  with  the  full  cooperation  of  the  program  offices.  This  exercise  also  marked  the 
first  employment  of  the  BMD  M&S  Integrated  Product  Development  Team  (IPDT)  concept 
with  the  objective  of  developing,  verifying,  and  validating  accurate,  authoritative  represen¬ 
tations  of  element  capabilities  in  the  System -Level  M&S  [Kad02a].  The  BMD  System- 
Level  M&S  V&V  program  incorporated  the  lessons-learned  from  [HSW+92,  BBG+99a, 
Bar02a,  CRC02a,  Bar02b,  CRC02b,  Obe02]  and  several  other  reports  cited  earlier,  and  de¬ 
veloped  centrally-managed  VV&A  processes  for  system-wide  events  [Kad02a]. 

J,  BMDS  SYSTEM  M&S  FIDELITY 

1.  Agency  M&S  Fidelity  Background  1986  -  2000 

The  Agency  adopted  a  “family  of  systems”  (FoS)  approach  to  describe  mutual  sup¬ 
port  over  a  hierarchy  ranging  from  high  resolution,  limited  scope  models  (e.g.,  physics 


level  codes)  at  the  base,  through  higher  level  overlapping  levels  of  models  with  lower  reso- 
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lution  [SW96,  BBB+96],  similar  to  the  Defense  Systems  Management  Collage  model 
[PMR94].  [SW96]  proposed  an  analytical  taxonomy  supporting  the  FoS  approach  with  six 
qualitative  levels:  system,  encounter,  engagement,  mission,  campaign,  and  global. 
[BBB+96]  took  a  different  approach,  basically  developing  a  two-dimension  array  with  four 
major  application  categories:  analysis,  test  and  evaluation,  exercises  and  wargaming,  and 
engineering  assessed  at  four  qualitative  levels  of  analysis:  campaign260,  force-on-force  en- 
gagement  ,  few-on-few  engagement  ,  and  special  domain"  ~  perfonnance. 

These  qualitative  approaches  were  not  sufficient  for  the  high-risk,  mission-critical 
missile  defense  domain.  A  later  1999  MITRE  Corporation-led  review  of  the  BMDO  M&S 
program  [RTM+99b]  addressed  two  major  BMDO  M&S  issues  central  to  this  dissertation. 
[RTM99b]  recommended  the  Agency: 

•  Identify  the  methodology  for  representing  systems  and  functions  (e.g.,  emplace¬ 
ment,  search  and  track,  engage,  missile  launch  /  fly  out,  tenninal  guidance,  intercept 
and  post-intercept  assessment)  critical  to  missile  defense  at  each  level  in  the  M&S 
hierarchy, 

•  Analyze  the  consistency  of  M&S  representations  of  functions  up  and  down  the 
M&S  hierarchy  [RTM+99b]. 

Another  related  internal  Agency  report,  [BBG+99b]  noted  the  quality  of  each  level 
of  simulation  depended  upon  the  quality  of  the  input  from  the  lower  level  simulation  or 
code  and  recommended: 

•  An  evaluation  of  the  required  levels  of  fidelity  between  program  office  high  fidelity 
simulations  and  factor-driven  campaign-level  simulation  tools, 

•  The  Agency  develop  an  agreed  upon  range  of  fidelity  for  the  kill-chain  components 
(e.g.,  sensors,  weapon  algorithms,  platforms,  C4I)  instead  of  a  set  fidelity  level  to 
close  the  “fidelity  gap”  between  different  organizations  [BBG+99b]. 


~60  Campaign:  Characteristics  include  aggregated  resolution,  theater-level  scope,  many-on-many  scenarios,  run  time  speed  many  times 
faster  than  real-time,  generally  lower  fidelity,  simulating  time  span  from  days  to  months,  includes  all  architecture  components  plus 
ground,  air,  and  naval  forces.  Addresses  logistics  factors:  resupply,  reload,  damage  and  repair,  and  weapon  inventory  issues.  Examines 
system  performance  impact  on  the  course,  duration  and  outcome  of  the  campaign  [BBB+96], 

Force-on-Force:  Characteristics  include  minimal  aggregation,  theater  level  scope,  many-on-many  scenarios,  medium-to-high  fidelity 
(e.g.,  treatment  may  be  mixed  with  three  of  freedom  (3  DoF)  modeled  for  some  aspects  while  way  points  may  be  used  for  other  trajecto¬ 
ries),  and  probably  does  not  play  intercept  or  end  game  performance.  Simulation  time  span  is  from  hours  to  a  few  days,  running  faster 
than  real  time  to  support  trade  studies,  and  usually  provides  data  for  campaign  models  [BBB+96], 

Few-on-Few:  Characteristics  one-on-one  engagements,  no  aggregation,  limited  scope,  high  system  detail  and  fidelity  of  representa¬ 
tion  for  system  under  examination.  Intercept  end  game  played,  simulation  time  form  a  span  of  minutes  to  a  few  hours,  run  time  faster 
than  real  time  to  facilitate  sensitivity  analysis,  examines  system  and  subsystem  performance.  Provides  data  for  Force-on-Force  models 
[BBB+96], 

~63  Special  Domain:  Configured  to  examine  particular  phenomena,  such  as  command  and  control,  battle  management,  communications, 
with  aggregation,  scope,  fidelity  of  representation,  and  run  time  requirements  dependent  on  specific  study  or  experiment  objectives 
[BBB+96], 
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2.  An  Evolving  M&S  Fidelity  Framework  2000  -  2003 

The  Agency  developed  a  framework  providing  decision-makers  with  a  background 
on  the  BMD  System-level  model  capabilities,  flexibility,  resolution,  accuracy,  and  preci¬ 
sion  (Figure  9-4)  relative  to  other  simulations  in  the  set  and  the  new  block-based  capability 
acquisition  policy  [Rum02c],  as  an  interim  method  pending  a  more  definitive  model 
[Gre02a].  In  October  2002,  the  MDA  Model  and  Simulation  Working  Group  (MSWG), 
composed  of  members  from  all  the  Agency  program  offices,  MDA  directorates  and  execut¬ 
ing  agents  established  a  fidelity  technical  working  group  (TWG)  with  the  following  char¬ 
ter: 

•  Define  M&S  fidelity  in  the  context  of  the  BMDS  System-level  M&S, 

•  Propose  a  time -phased  schedule  for  implementing  BMD  fidelity  standards, 

•  Determine  the  requirement  for  multi-resolution  capabilities  [Gre02b]. 
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Figure  9-4.  MDS  System-Level  Model  and  Simulation  Capability  (After  [TEM03]) 
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One  of  the  early  product  deliverables  from  the  BMDS  MSWG  Fidelity  technical 
working  group  (Figure  9-5)  provides  the  methodology  the  group  selected  to  support  their 
tasks.  The  TWG’s  initial  effort  [Gre02b]  addressed  the  BMDS  kill-chain  detection  func¬ 
tion,  sub-functions  (e.g.,  search,  detect,  initiate  track,  track)  and  the  respective  fidelity  at¬ 
tributes  of  the  BMD  sensor  family  of  radar  detection  models  as  they  relate  to  the  BMDS 
System  Capability  Specification,  a  major  BMD  system  engineering  specification.  The 
[WAD+02]  approach  also  addresses  another  major  Department  systemic  deficiency  where 
major  M&S  initiatives  and  the  Department’s  system  engineering  processes  lack  a  clear  in¬ 
tegrated  relationship,  linkage,  and  common  focus. 
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Figure  9-5.  Example  of  Radar  Fidelity  in  Core  Models  (From  [TEM03]) 
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3.  BMD  System-Level  M&S  Radar  Detection  Models 

There  are  several  critical  functions  and  sub-functions  identified  in  the  BMDS  kill- 
chain  framework  (see  Figure  9-2)  required  for  the  successful  execution  of  the  BMDS  mis¬ 
sion.  However,  in  this  dissertation,  we  address  only  a  subset  of  the  detection  function  at  a 
very  high  level  to  compare  and  contrast  the  development  designs  for  radar  detection  models 
in  the  five  BMDS  system-level  simulations.  These  radar  model  representations  form  pat¬ 
terns  used  later  as  the  foundation  for  proposals  on  product  lines  and  components. 

The  CAPS  simulation,  considered  “low”  fidelity,  provides  Radar  Detection  Models 
with  three  basic  detection  capabilities: 

•  Constant  range,  referred  to  as  “cookie  cutter”, 

•  Scaled  equation  with  reference  parameters, 

•  Full  range  equation  [SpaOO] 

CAPS  also  provides  various  fidelities  supported  by  four  different  threshold  options: 

•  Deterministic,  using  a  standard  signal-to-noise  threshold  defined  by  the  user, 

•  Probabilistic,  using  a  random  probability  of  detection  threshold, 

•  Deterministic,  using  a  probability  of  detection  (Pd)  defined  by  the  user, 

•  Deterministic,  with  cumulative  Pd  threshold  set  by  the  user  [SpaOO], 

The  optional  degrees  of  flexible  model  methods  and  fidelity  may  appear  as  an  unexpected 
capability  in  a  highly  aggregated  simulation  of  CAPS  class,  except  the  detect  function  of 
the  BMD  kill  chain  is  critical  to  the  credibility  of  all  BMD  system-level  M&S.  Parameters 
[SpaOl]  maintained  in  database  files  support  most  CAPS  functions. 

The  MDWAR  simulation,  considered  “low-medium”  fidelity,  employs  a  generic  ra¬ 
dar  sensor  model  for  detecting  and  tracking  simulated  threat  targets,  integrated  with  the 
Parallel  Discrete  Event  Simulation  (PDES)  engine  [TRW02a].  The  model  supports  all  the 
events  needed  to  create  a  generic  sensor  model  and  certain  specific  radars,  with  design  con¬ 
siderations  driven  by  a  need  for  a  fast  generic  sensor  simulation  modeling  components 
common  to  all  types  of  sensors  [TRWOOb].  The  MDWAR  real-time  radar  simulator,  op¬ 
timized  for  large,  real-time  C2  wargame  simulations  may  be  configured  to  represent  a  num¬ 
ber  of  different  radars,  with  a  design  goal  of  tracking  1,200  air  breathing  threats  (e.g.,  air¬ 
craft,  cruise  missiles)  with  100  radars  while  maintaining  wall-clock  time  [TRW02b]. 
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The  MDWAR  design  objectives  support  the  requirement  to  provide  users  with  real¬ 
time  radar  simulations  driving  fifty  to  seventy- five  player/observer  graphic  workstations  in 
very  large  human-in-the-loop  wargame  scenarios  [TRW02b].  In  order  to  achieve  the  re¬ 
quired  perfonnance,  the  radar  sensor  model  depends  on  the  following  assumptions: 

•  All  sensor  fidelity  enhancements  must  be  selectable  by  .par  (e.g.,  parameter  files)  so 
they  can  be  switched  off  if  not  required, 

•  MDWAR  models  perfect  communications  between  the  sensor  and  the  battle  man¬ 
ager,  as  opposed  to  perception-based  methods  used  in  other  simulations, 

•  MDWAR  models  perfect  weather  without  adverse  conditions  that  may  add  to  the 
fog  and  friction  of  war, 

•  MDWAR  does  not  model  chaff  clouds  or  jammers  [TRW02b]. 

MDWAR  does  not  model  radar  resolution,  but  provides  perfect  radar  resolution  in 
which  every  truth  target,  if  detected,  generates  a  radar  return.  However,  discrimination  is 
imperfect,  generated  by  a  time -based  probability  model  [TRW02b].  The  radar  sensor 
model  has  unlimited  sensor  resources,  except  real-time  constraints  for  keeping  up  with  the 
wall-clock  time;  employs  radar  cross-section  (RCS)  data  from  a  classified  data  table  based 
on  target  type,  radar  frequency,  and  polarization;  employs  a  Swerling  target  model  (e.g., 
cases  0,1, 2, 3, 4);  and  computes  Pd  on  a  random  draw  to  detennine  if  detection  happens  or 
not  [TOM99,  TRW02b].  MDWAR  uses  truth-based  algorithms,  versus  a  probability  model 
[AGJ95]  to  detennine  a  hit  or  miss  for  kill  assessment  [TRW02], 

EADSIM,  considered  “medium”  fidelity,  also  provides  significant  flexibility  and 
options  for  radar  representations,  including  complex  models  supporting  both  deterministic 
and  probabilistic  radar  range  detection  [TOM97].  Signature  modeling  is  activity  dependent 
and  includes  user-specified  functions  for  frequency,  wavelength,  aspect  angle,  with  the  user 
specifying  the  granularity  of  the  tabular  data  [TOM97,  TBE02a].  The  radar  sensor  model 
may  use  a  radar  range  equation,  with  jamming  to  detect  targets,  with  the  radar  sensor  class 
using  a  detection  probability  employing  signal-to-noise  (SNR)  ratio,  RCS,  and  bum- 
through  calculations  [TBE02b] . 

EADSIM  employs  a  detection  gate  criteria  as  a  means  to  model  sensor  detection 
limitations  with  a  four  gate  criteria: 

•  Sensor-to-target  range, 

•  Absolute  target  speed, 
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•  Sensor-to-target  range  rate, 

•  Target  altitude  [TBE02b]. 

The  four  gate  criteria  share  common  test  criteria  with  a  region  defined  by  minimum  and 
maximum  values,  and  no  detections  possible  outside  the  defined  region  [TBE02b].  EAD- 
SIM  defines  RCS  in  one  of  three  ways:  uniform  RCS,  a  single  value  specified  for  a  system 
or  weapon  type;  roll  symmetric  RCS  tables,  defined  by  frequency;  or  laterally  symmetric 
RCS  tables,  normally  employed  with  aircraft  and  cruise  missiles,  when  the  right  and  left 
sides  of  the  system  are  mirror  images  [TBE02b]. 

EADSIM  supports  radar  sensors  at  different  levels  of  fidelity  including  a  simple 
sensor  approximating  the  to-level  performance  of  the  radar  with  basic  search,  detect,  and 
track  functions;  a  compound  sensor  supporting  multifunction  radars  and  advanced  func¬ 
tions  including  multiple  autonomous  search  sectors,  cued  search,  track,  interceptor  support 
functions,  resource  management,  sensor  constraints  and  kill  assessment  [TOM97].  In 
EADSIM,  target  detection  includes  specific  sensor-to-target  detection  tests  including: 

•  There  is  an  unobstructed  view  of  the  target, 

•  There  is  an  unobstructed  line-of-sight  between  the  target  and  the  sensor, 

•  The  target  is  in  the  sensor’s  field  of  view  [TOM97]. 

Specific  detection  tests  may  be  either  detenninistic  comparing  computed  signal-to- 
interference  ratios  with  a  user-specified  variable,  or  probabilistic  with  Swerling  target  fluc¬ 
tuation  models  providing  a  computed  probability  of  detection  with  a  random  draw  to  de¬ 
termine  detection  [TOM97,  TBE02b].  Both  methods  support  the  effects  of  clutter,  multi- 
path,  atmospheric  attenuation,  and  jamming  on  the  radar  sensor  [TB02b].  In  all  cases 
EADSIM  employs  digital  terrain  models  to  check  for  terrain  masking  and  inter-visibility 
issues  [TOM97]. 

EADTB,  considered  a  “medium-high”  fidelity  simulation,  supports  sensor  models, 
which  detect  objects  through  the  measurement  of  energy,  reflected  or  emitted  by  those  ob¬ 
jects  (e.g.,  the  sensed  platform  supplies  the  signature,  and  not  the  sensing  platform).  The 
sensed  object  provides  the  signature  on  request  based  on  the  sensing  geometry  [RayOlb]. 
The  user  defines  sensor  perfonnance  (e.g.,  power,  gain,  target  signature  characteristics,  en- 
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vironmental  effects)  input  variables,  which  the  sensor  algorithm  computes  to  detect  targets 
[Ray02a]. 

Although  EADTB  models  both  active  and  passive  sensors  including  electro-optic  / 
infrared  passive  sensors,  and  RF  passive  sensors,  radar  detection  models  are  active  sensors 
that  transmit  radio  frequency  (RF)  energy  and  measure  its  reflection  from  the  target  body 
[Ray02a].  EADTB  sensor  models  include  separate  methods  for  detection  and  tracking  and 
may  include: 

•  Scanning, 

•  Jamming, 

•  Line-of-sight  obscuration, 

•  Antenna  patterns, 

•  Sensitivity  and  processing, 

•  Sensor  beam  or  field  of  view  [RayOlb]. 

The  level  of  detail  in  an  EADTB  sensor  model  varies  from  a  functional  sensor  in¬ 
corporating  range,  target  type,  and  search  volume;  to  a  high-detail  sensor  with  many  input 
sensors  to  model  gain  patterns,  polarization,  Doppler  spreads,  ambiguous  ranges,  pulse 
compensation  ratios,  and  main-  or  side-lobe  jamming  [Ray02a].  The  modular  sensor 
model  includes  four  subcomponents:  scanner,  transmitter,  receiver,  and  data  processor  and 
the  management  of  these  resources,  while  the  radar  model  simulates  different  radar  func¬ 
tions  (e.g.,  search,  track,  acquisition)  and  different  types  of  radars  (e.g.,  mechanical,  phased 
array)  [Ray02a]. 

EADTB  computes  the  probability  of  detection  as  a  function  of  the  probability  of  de¬ 
tection  versus  SNR  distribution  and  the  difference  between  the  SNR  and  the  detection 
threshold,  followed  by  a  random  draw  to  detennine  target  detection  [Ray02a].  EADTB 
sensors  operate  in  explicitly  defined  modes  supporting  multiple  scan  modes  and  track 
modes  and  accommodate  three  classes  of  data:  static,  dynamic,  and  mode-dependent 
[BBB+96].  In  addition,  EADTB  may  model  different  RF  phenomena,  with  over  180  types 
of  radar  data  input  possible. 

MDSE  currently  tests  the  BMDS  at  the  FoS  level  (e.g.,  interoperability),  as  opposed 
to  the  system  or  subsystem  level,  which  establishes  the  required  level  of  fidelity  and  accu¬ 
racy  for  the  three  embedded  model  categories:  BMC2,  launcher  and  fly-out  models,  and 
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sensor  front  end  models  [McQ97].  The  MDSE  sensor  front-end  model  provides  accurate 
detection,  tracking  positions,  and  timelines  with  the  initial  detect  function  of  the  kill  chain 
required  to  be  accurate  within  +/-  10%  [Too94].  MDSE  provides  the  RCS  for  missiles  as  a 
function  of  aspect  angle  (e.g.,  a  measurement  from  missile  nose  to  tail),  required  for  initial 
detection  and  tracking  results  [McQ97].  MDSE  sensor  models  generate  realistic  track  posi¬ 
tion  fluctuations  for  covariance  matrices  supporting  the  development  of  Tactical  Digital 
Information  Link  (TADIL)  message  standard  [DoD97]  J3.6  messages.  Although  MDSE 
radar  models  are  not  detailed  enough  to  perform  the  classification  or  discrimination  func¬ 
tions,  they  are  sufficient  for  use  in  determining  the  status  or  reporting  responsibility  (R2) 
message  traffic  [Too94]. 

K.  BMD  SYSTEM-LEVEL  M&S  REPRESENTATION  HETEROGENITY 


A  review  of  Table  9-1  through  Table  9-16  reveals  several  specific  contributing 
causes  for  M&S  representation  heterogeneity  issues  in  the  five  BMDS  System-Level  M&S, 
and  possible  contributing  causes  for  M&S  representation  heterogeneity  issues  in  the  De¬ 
partment’s  large-scale  legacy  M&S  including: 

•  The  absence  of  a  specific  system  representation  for  existing  systems  within  a  single 
simulation, 

•  The  absence  of  a  specific  system  representation  for  existing  systems  within  a  se¬ 
lected  set  of  federated  simulations, 

•  The  absence  of  specific  system  representations  for  different  configurations  of  the 
same  existing  system  within  the  same  simulation  (e.g.,  THAAD  PDRR  system  and 
the  THAAD  EMD  system), 

•  The  absence  of  specific  system  representations  in  the  selected  set  of  federated  simu¬ 
lations  for  the  same  existing  system/platform  (e.g.,  Aegis  Weapon  System)  with  dif¬ 
ferent  or  unique  configurations  (e.g.,  SM-3  Block  1  A,  Navy  Theater  Wide  (NTW), 
and  Navy  LEAP  ALI), 

•  Temporal  heterogeneity- 

o  CAPS  is  a  discrete-event  simulation  and  a  hybrid  event-driven,  time-stepped 
constructive  simulation  [SpaOO],  In  the  hybrid  event-driven,  time-stepped 
constructive  simulation  mode,  time-compression  algorithms  reduce  run  time 
in  scenarios  with  several  periods  of  intense  activity,  separated  by  long  peri¬ 
ods  of  relative  inactivity  prior  to  the  next  event.  Using  time  compression, 
CAPS  is  able  to  run  an  80-day  theater-level  campaign  in  approximately  an 
hour,  depending  on  the  available  machine  resources, 
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o  EADSIM  is  both  event-driven  and  time-stepped,  in  a  hybrid  approach  where 
the  C3I  processes  are  event  driven  and  all  other  processes  are  time-stepped, 

•  Geodetic  heterogeneity  inconsistencies  cited  in  Table  9-15  are  a  contributing  cause 
of  heterogeneous  system  representation  anomalies,  including  substantive  interop¬ 
erability  discussed  in  Chapter  IV.  The  requirement  to  accurately  model  the  BMD 
on  worldwide  basis  dictates  a  standard  geodetic  environment,  accurately  established 
in  all  System-Level  M&S. 

L.  BMDS  SYSTEM-LEVEL  M&S  DEVLOPMENT  DEMOGRAPHICS 

Table  9-18  provides  the  development  demographics  for  the  five  BMD  System- 
Level  simulations.  Official  development  for  all  five  simulations  averaged  slightly  over 
three  years  and  ranged  from  two  to  five  years  for  the  first  software  release.  MDA  execu¬ 
tive  agents264  in  Huntsville,  AL,  and  Colorado  Springs,  CO  manage  four  of  the  systems  de¬ 
veloped  by  contractors.  The  number  of  developers  cited  in  Table  9-18  indicates  the  aver¬ 
age  annual  simulation  software  development  level  of  effort  (LOE)  allocated  for  these  sys¬ 
tems  for  the  fiscal  years  2000-2003,  with  the  exception  of  EADSIM.  The  EADSIM  pro¬ 
gram  manager  determines  requirements  and  accepts  resources  from  over  three  hundred  or¬ 
ganizations.  Only  three  of  the  thirty-six  EADSIM  developers  noted  in  Table  9-18  directly 
support  MDA  requirements. 

The  simulation  software  developer  metrics  for  the  missile  domain  M&S  is  an  im¬ 
portant  metric,  as  is  the  location  of  the  major  development  efforts.  The  Fletcher  Report 
[FMA+84]  provided  strong  justification  for  mature  software  development  organizations 
with  strong  missile  domain  expertise  employing  simulations  as  a  design  tool  noting: 

Specifying,  generating,  testing,  and  maintaining  the  soft¬ 
ware  for  a  battle  management  system  will  be  a  task  that  far 
exceeds  in  complexity  and  difficulty  any  that  yet  has  been 
accomplished  in  the  production  of  civil  or  military  software 
systems  [FMA+84]. 

The  number  of  software  developers  and  maintainers  for  the  missile  defense  domain 
M&S  software  identified  in  Table  9-18  identify  sources  of  the  current  domain  expertise  and 
market  presence  required  to  meet  the  software  challenges  cited  by  references  earlier  in  the 
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Missile  defense  executive  agents  are  government  agencies  in  locations  including  Huntsville,  AL.,  Colorado  Springs,  CO.,  and  San 
Diego,  CA„  managing  MDA-directed  and  funded  projects.  Contactors  develop  most  of  the  products  managed  by  the  executive  agents. 
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chapter,  and  earlier  missile  defense  studies  including  the  Fletcher  Report  [FMA+84],  the 
follow-on  Eastport  Study  Group  report  [CCL+85],  and  Congressional  studies  by  the  Office 
of  Technology  Assessment  [SCC+88].  The  co-location  of  the  simulation  software  devel¬ 
opers  with  the  missile  defense  program  offices  and  their  software  developers  in  Huntsville, 
AL.,  and  with  the  developers  of  the  battle  management  software  in  Colorado  Springs,  CO., 
supported  by  a  distributed  network  of  missile  defense  expertise  play  a  critical  role  in  the 
M&S  evolutionary  acquisition  strategy  supporting  the  BMDS. 
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Table  9-18.  BMD  System-Level  M&S  Demographics 


The  Department  requires  process  maturity  from  the  software  acquirers  and  the 
software  developers.  The  BMDS  government  personnel  acquiring  the  simulation  software 
products,  must  understand  the  missile  defense  system  evolutionary  acquisition  strategy 
[Ald02d]  and  also  develop  and  define  the  applicable  methods  (e.g.,  Software  Acquisition- 
Capability  Maturity  Model  (SA-CMM),  or  a  similar  methodology  to  determine  and  report 
process  adherence  and  performance  metrics  [AS03].  Similarly,  the  contractors  serving  as 
the  builders  and  vendors  for  the  core  simulation  software  assets  must  have  the  software  de¬ 
velopment  and  performance  skills  [Gan99]  to  understand  and  translate  the  underlying  com¬ 
plex  science,  engineering,  and  technology  into  the  missile  defense  domain  M&S  software 
at  a  minimum  of  SEI  CMM  Level  3,  or  its  equivalent.  The  SEI  Capability  Maturity  Model 
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(CMM)  assessment  for  three  of  the  five  software  development  organizations  for  MDWAR, 
EADSIM,  and  MDSE  is  CMM  level  3,  while  a  fourth  developer  for  EADTB  is  pursuing 
ISO  9000  certification. 

M.  ANALYSIS  OF  BMDS  M&S  SOFTWARE  STATE  FACTORS 

Although  these  five  simulations  arrived  at  their  current  state  by  different  circum¬ 
stances  and  development  models,  they  share  several  common,  important  characteristics. 
First,  as  noted  previously  in  the  M&S  synopsis  for  each  simulation,  these  simulation  tools 
meet  a  unique  requirement  (e.g.,  Human-in-the-loop,  Hardw are-in- the-loop,  architecture 
development)  for  the  development  of  the  missile  defense  system.  Second,  there  is  a  long 
lineage  of  experienced  stakeholders,  a  distributed  organizational  cohort  of  users,  develop¬ 
ers,  warfighters  who  understand  the  underlying  missile  defense  technologies,  have  devel¬ 
oped  relationships  within  the  community,  interact  often  on  studies,  exercises,  or  analyses, 
and  share  the  same  common  mission- focus.  Third,  these  simulations  produce  results 
deemed  credible  enough  for  major  Department  decision-makers  to  support  the  authoriza¬ 
tion  and  allocations  of  continued  Department  investments  for  their  evolutionary  develop¬ 
ment.  Fourth,  these  simulation  systems  represent  the  repositories  of  missile  defense  do¬ 
main  expertise,  intellectual  capital,  and/or  presence  in  the  marketplace.  Fifth,  these  simula¬ 
tions  are  missile  defense  core  assets  and  repositories  for  missile  defense  requirements, 
knowledge  and  insights  that  may  not  be  well  documented  or  understood  elsewhere. 

Fastly,  in  an  analogous  sense,  these  five  legacy  simulation  systems  and  support  en¬ 
vironments  survived  a  Darwinian  winnowing  process  as  many  other  Department  simulation 
systems  atrophied,  expired,  or  experienced  a  stasis  of  resources  during  the  1990s.  The  co¬ 
hort  of  organizational  sponsors,  users,  collaborators,  and  developers  for  these  five  simula¬ 
tion  systems  shared  a  parallel  sociological  process  as  a  distributed  organizational  constitu¬ 
ency  and  have  at  this  point  satisfied  what  Maslow  would  have  identified  as  the  physiologi¬ 
cal  need  of  survival.  The  distributed  organizational  constituency  or  cohort  of  supporters, 
collaborators,  and  stakeholders,  nonexistent  when  the  simulations  were  first  fielded,  ex¬ 
panded  their  knowledge-base  concurrently  over  time  with  the  maturation  of  the  simula¬ 
tions,  and  contributed  significantly  to  the  systems  development  and  longevity  There  is  an- 
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other  important  consideration.  All  five  system-level  simulation  systems  completed  the  dif¬ 
ficult  early  life  cycle  phases  including  the  problematic  requirements  generation  process  and 
the  continuous  justification  for  funding. 

Working  within  the  Departmental  transformation  process,  and  supporting  an  evolu¬ 
tionary  block  acquisition  strategy,  the  missile  defense  domain  system  engineering  process, 
simulation  software  engineering  capability,  supporting  organizations,  and  processes  need  to 
concurrently  mature  in  an  extremely  complex  and  technically  demanding  environment. 
Furthermore,  when  compared  to  less-complex  systems,  critical  differences  in  the  missile 
defense  software  quality  requirements  exist,  which  would: 

•  Permit  less  opportunity  for  human  intervention, 

•  Have  to  handle  more  objects  in  its  battle  space, 

•  Have  to  manage  a  larger  battle  space, 

•  Use  different  weapons  and  sensor  technology, 

•  Contain  vastly  more  elements, 

•  Have  more  serious  consequences  of  failure, 

•  Have  to  operate  in  a  nuclear  environment, 

•  Be  under  active  attack  by  the  enemy,  and 

•  Be  useless  if  it  failed  catastrophically  during  its  first  battle  [SCC+88], 

A  review  of  the  Agency’s  funding  documents  for  the  selected  core  M&S  revealed 
an  inconsistent  funding  profile  before  2000  for  the  reasons  noted  earlier  in  the  chapter.  In 
addition,  resolution  of  potential  Year  2000  problems  [Gre97,  WG99]  and  the  Department’s 
mandate  for  HLA  interoperability  expended  funding  resources  eannarked  for  simulation 
development  during  the  1990’s  [ISE99].  In  the  future  the  core  M&S  program  appears  as  a 
relatively  stable  program  according  to  the  Department’s  Future  Years  Defense  Plan  and  the 
Program  Objective  Memorandum  documents,  with  level  and  consistent  funding  levels  pro¬ 
jected  through  2007.  However,  with  no  new  projected  funding  additions,  future  simulation 
software  engineering  improvements  will  compete  for  a  finite  set  of  the  Agency’s  resources. 

N.  ANALYSIS  OF  BMDS  M&S  SOFTWARE  DEVELOPMENT  PORTFOLIO 

Table  9-19  identifies  the  programming  languages  and  the  associated  source  lines  of 
code  (SLOC)  of  the  five  software-intensive  BMD  System-Level  Model  and  Simulations. 
The  BMD  System-Level  M&S  comprise  a  total  of  nearly  8.5  million  SLOC.  This  corpus 
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of  code  is  a  domain  capital/core  asset  of  the  BMD  system  portfolio.  A  few  programming 
languages  including  Ada,  C,  C++,  and  FORTRAN  equate  to  approximately  ninety-three 
percent  of  the  total  SLOC.  The  Ada  (28%)  and  C  (32%)  programming  languages  are  sixty 
percent  of  the  total  SLOC  for  the  five  simulations,  a  metric  heavily  influenced  by  the  2.25 
million  SLOC  of  Ada  code  resident  in  the  EADTB  simulation.  The  C  or  C++  languages 
are  approximately  fifty-live  percent  of  the  total  portfolio  SLOC. 

A  review  of  Table  9-19  reveals  less  than  two  percent  of  the  Ada  code  written  by 
CMM  level  3  software  developers.  Conversely,  the  three  CMM  level  3  software  develop¬ 
ers  identified  in  Table  9-19,  wrote  approximately  4.2  million  SLOC,  or  nearly  fifty  percent 
of  the  total  8.5  million  SLOC  in  C  or  C++.  In  addition  the  C  and  C++  programming  lan¬ 
guages  are  also  the  most  commonly  used  programming  languages,  with  a  significant  pres¬ 
ence  in  four  of  the  five  simulations. 

Viewed  from  the  simulation  software  perspective,  MDSE  (35%),  EADTB  (34%), 
and  EADSIM  (20%)  contain  nearly  ninety  percent  of  the  total  8.5  million  SLOC  of  the 
five-model  portfolio,  with  EADTB  and  MDSE  approaching  the  three  million  SLOC  mile¬ 
stones.  CAPS,  the  smallest  BMD  System-Level  M&S  employs  mostly  FORTRAN,  C  and 
C++.  The  C++  and  JAVA  development  environments  support  the  MDWAR  effort.  EAD¬ 
SIM  described  by  [TOM97]  as  “object-like  nature”  and  written  primarily  in  C  and  C++, 
with  some  FORTRAN,  has  the  highest  population  of  other  COTS  and  proprietary  lan¬ 
guages.  EADTB  is  an  event-driven  simulation  written  primarily  in  Ada,  with  some  com¬ 
ponents  developed  in  C  and  JAVA,  and  several  other  languages.  MDSE  is  the  largest  lan¬ 
guage  development  and  maintenance  environment  including  C++,  C,  FORTRAN,  JAVA, 
and  Ada,  due  to  its  distributed  development  environment.  Oracle  relational  database  sys¬ 
tems  house  MDSE  and  EADTB. 

In  many  respects  the  development  cycle  of  these  five  simulations  listed  in  Table  9- 
19  mirror  the  Department’s  software  development  experience  since  1985.  CAPS  and 
EADSIM  were  developed  in  an  incremental,  evolutionary  manner  from  an  existing  capabil¬ 
ity  at  relatively  low  and  inconsistent  funding  levels,  separate  from  the  Department’s  acqui¬ 
sition  thresholds  requiring  the  use  of  Ada.  With  the  exception  of  the  use  of  FORTRAN, 
these  systems  developed  around  the  languages  du  jour,  primarily  C  and  C++. 
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Conversely,  EADTB  and  MDSE  were  relatively  large  systems  with  the  need  to  jus¬ 
tify  significant  amounts  of  Department  appropriated  funding,  which  triggered  the  imple¬ 
mentation  of  the  Ada  language.  Approximately  seventy-eight  percent  of  EADTB  remains 
in  the  Ada  development  environment,  with  a  significant  presence  in  C  and  a  growing  body 
of  JAVA  code.  In  MDSE  the  amount  of  Ada  code  equates  to  less  than  five  percent  of  the 
MDSE  system  total,  while  C  and  C++  correspond  to  approximately  sixty- five  percent  of 
the  MDSE  portfolio,  followed  by  a  twenty-eight  percent  FORTRAN  population,  the  largest 
FORTRAN  presence  in  the  five  simulations  under  study. 


BMDS  SYSTEM-LEVEL  M&S  LANGUAGES  /  SLOC  -  2002 


MDL / LANG 

ADA 

C 

C++ 

FORTRAN 

JAVA 

Other-65 

Total 

CAPS 

600 

650 

750 

2000 

MDWAR 

872,434 

47,335 

919,769 

EADSIM 

1,338,481 

99,705 

34,234 

225,933 

1,698,353 

EADTB 

2,250,000 

490,000 

80,000 

80,000 

2,900,000 

MDSE 

146,800 

891,600 

1,051,000 

837,300 

47,000 

2,973,700 

TOTAL 

2,396,800 

2,720,681 

2,023,789 

872,284 

174,335 

305,933 

8,493,822 

Table  9-19. 


BMD  System-Level  M&S  SLOC  -  2002  (After  [TEM03]) 


O.  BMDS  M&S  SOFTWARE  INTEROPERABILITY 


BMDS  System-Level  M&S  distributed  interoperability  capabilities  includes  HLA, 
DIS,  and  ALPS.  Table  9-20  illustrates  the  distributed  interoperability  status  of  the  five 
simulations  under  study.  Four  of  the  five  simulations,  for  an  eighty  percent  compliance 
rate,  are  HLA-compliant,  with  MDSE,  the  agency  hardware-in-the-loop  tool  as  the  sole  ex¬ 
ception.  A  review  of  the  simulations  supporting  the  DIS  standard  also  reveals  an  eighty 
percent  compliance  rate,  with  CAPS  being  the  only  simulation  under  review  lacking  DIS 
support.  From  the  interoperability  perspective,  all  five  simulations  are  technically  interop¬ 
erable  with  either  the  HLA  or  DIS  protocols,  and  sixty  percent  of  the  simulations,  including 
MDWAR,  EADSIM,  and  EADTB  are  both  HLA  and  DIS  compliant.  EADSIM  is  the  only 
one  of  the  five  simulations  supporting  all  three  interoperability  standards,  while  CAPS  sup¬ 
ports  only  HLA.  Overall,  the  systems  under  study  achieved  eighty  percent  for  HLA  and 
DIS  interoperability,  twenty  percent  for  the  ALPS  standard,  with  an  overall  distributed 


Other  languages  include  a  mix  of  COTS  and  proprietary  languages:  Perl,  SQL,  HTML,  C30,  RSRC,  H  [TEM03] 
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simulation  capability  of  sixty  percent  when  considering  all  three  M&S  distributed  M&S 
protocol  standards. 


BMDS  SYSTEM-LEVEL  M&S  DISTRIBUTED  SIMULATION  CAPABILITY 

INTEROPERABILITY 

CAPS 

MDWAR 

EADSIM 

EADTB 

MDSE 

% 

HI.  A 

Y 

Y 

Y 

Y 

N 

80% 

DIS 

N 

Y 

Y 

Y 

Y 

80% 

ALPS 

N 

N 

Y 

N 

N 

20% 

#  Avail  /Total/  %  Avail 

1/3/33% 

2  /  3  /  66% 

3/3/100% 

2  /  3  /  66% 

1/3/33% 

60% 

[SpaOO, 

[CM99, 

[BAC+95, 

[BBG+96, 

[Too94, 

SpaOl, 

TOM99, 

TOM97, 

RayOla, 

McQ97, 

Citation  from 

BBG+99f] 

TRWOOa, 

BBG+99e, 

Rayo2a, 

TEM00] 

TRWOOb, 

TBE02a, 

RayOlb] 

TRW02b] 

TBE02b| 

Tab 


e 


9-20. 


BMD  System-Level  M&S  Interoperability  Status 


P.  EDUCATION  ENABLERS  SUPPORTING  M&S  CREDIBILITY 

A  few  years  ago  a  television  commercial  showed  a  scene  where  a  company  execu¬ 
tive,  after  receiving  a  presentation  on  the  problems  facing  his  company,  asked  the  same 
team  to  help  resolve  the  issues.  In  response  the  presenters  noted  they  only  identified  prob¬ 
lems,  and  did  not  fix  them.  The  Department’s  history  with  resolving  persistent  systemic 
software  issues  is  analogous  to  this  scenario.  The  Department  has  done  a  thorough  job 
identifying  the  many  issues  adversely  affecting  simulation  software  credibility.  There  have 
been  many  significant  policies,  programs,  processes,  and  procedures  implemented  to 
counter  these  issues,  although,  Department  simulation  software  still  lacks  overall  credibil¬ 
ity. 

One  of  the  underlying  reasons  for  a  lack  of  credible  simulations  is  the  Department’s 
lack  of  depth,  understanding,  experience,  and  maturity  with  disciplined  software  engineer¬ 
ing.  We  believe  a  critical  pre-condition  for  resolving  this  shortcoming  is  the  need  for  con¬ 
tinuing  software  engineering  education  [Tuc02]  within  the  Department,  acknowledging  the 
software  engineering  field  is  still  coalescing  [Boe02,  CT02,  HH02a,  HH02b,  SBM02, 
SHM02,  UHC02],  In  a  similar  situation  during  the  1980’s  the  Department  identified  the 
need  to  improve  the  professional  development  of  an  industrial-age  acquisition  workforce 
and  implemented  the  Defense  Acquisition  Workforce  Improvement  Act.  Understanding 
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the  changing,  challenging  dynamics  represented  by  today’s  infonnation-age  working  envi¬ 
ronment,  and  the  future  demands  of  continuous  transformation,  the  Department  instituted  a 
Continuous  Learning  Policy  [Ald02f],  acknowledging  the  need  to  operate  as  a  continuous 
learning  community,  continually  striving  to  improve  professional  knowledge  and  perform¬ 
ance  in  the  Acquisition  workforce. 

Today,  the  Nation  has  an  infonnation-based  economy  and  the  Department  has  an 
infonnation-based  national  military  strategy  based  on  software.  Unfortunately,  there  are 
few  indications  that  the  Department  has  developed  the  new  capabilities  to  define,  develop, 
acquire,  and  manage  the  myriad  of  complexities  involved  in  software-intensive  systems. 
The  Department  recently  acknowledged  this  shortfall  and  developed  a  program  [Ste02b]  to 
enable  selected  members  of  the  Department’s  military  and  civilian  workforce  earn  master 
or  doctoral  degrees  in  the  relevant  scientific,  technical,  and  management  academic  disci¬ 
plines  supporting  information  assurance. 

In  a  similar  program,  Dr.  Patricia  Sanders,  the  current  MDA  System  Executive  Of¬ 
ficer  and  former  BMDO  Deputy  for  Test,  Simulation,  and  Evaluation,  initiated  the  BMDO 
Software  Engineering  Improvement  Program  in  2000,  in  conjunction  with  the  Naval  Post¬ 
graduate  School’s  (NPS)  Software  Engineering  Distance  Learning  program,  pioneered  by 
Professor  Luqi.  The  NPS  Software  Engineering  Distance  Learning  program  provides 
Software  Engineering  programs  at  the  master  and  doctoral  level,  for  Department  software 
practitioners  to  study,  research,  and  advance  software  engineering  principles  and  technol¬ 
ogy  vital  to  Department  researchers  and  program  managers  [NPS02].  The  BMDO  Soft¬ 
ware  Engineering  Improvement  Program  complemented  the  Agency’s  Acquisition  Certifi¬ 
cation  Program  supporting  employee  Level  III  Defense  Acquisition  Corps  certification 
within  eighteen  months  of  arrival  and  an  eighty-hour  continuous  learning  program  for  the 
Level  III  workforce  every  two  years. 

The  melding  of  infonnation  technology  employed  in  the  NPS  distance-learning 
model,  with  the  collaborative  ability  of  a  distributed  learning  process  shared  by  multiple 
major  Department  acquisition  programs  in  different  domains  located  around  the  Nation 
(e.g.,  the  Army’s  Tank  and  Automotive  Command,  (TAACOM)  in  Michigan,  the  Navy’s 
Space  and  Warfare  Command  (SPAWAR)  in  California,  and  the  Missile  Defense  Agency 
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(MDA)  in  Washington,  DC),  employing  the  adult,  cooperative  learning  model  (e.g.,  study- 
and  work-related  research),  working  closely  with  and  mentored  by  the  NPS  Software  Engi¬ 
neering  faculty  appears  to  be  exactly  the  type  of  education  Trans  formation  model  the  De¬ 
partment  needs.  Software  engineering  research  supported  by  the  NPS  Distance  Learning 
model  also  has  a  reciprocating  function,  continually  updating  the  academic  community 
with  real-world  research  issues,  and  providing  a  source  for  the  latest  research  to  resolve 
current  Department  software  engineering  challenges.  A  recent  example  of  this  successful 
paradigm  was  the  publication  of  an  NPS  Software  Engineering  Distance  Learning  Master’s 
thesis  Conceptual  Framework  Approach  for  Systems-of-Systems  Software  Developments 
[Caf03]  in  support  of  the  BMDS  evolutionary  acquisition  strategy266. 

Q.  SUMMARY 

Chapter  IX  provided  an  overview  of  the  BMDS  M&S  domain  and  included  domain 
background,  the  domain  M&S  hierarchy,  a  BMDS  System-Level  M&S  synopsis,  M&S 
demographics,  and  a  review  of  the  missile  defense  system  representations  populating  the 
five  BMD  System-Level  simulations  under  study  in  this  dissertation.  The  BMD  domain 
overview  section  provided  the  Agency  background  and  highlighted  the  significant  role  De¬ 
partment  organizational  changes  played  in  the  current  status  of  BMD  System-Level  M&S. 

Building  on  the  organizational  outline  and  M&S  domain  overview,  the  chapter  con¬ 
tinued  with  a  status  review  on  the  domain  implementation  of  the  Department’s  policies  for 
establishing  credibility,  including  a  concise  VV&A  background  for  each  of  the  BMD  Sys¬ 
tem-Level  M&S.  Summaries  of  the  other  M&S  in  the  domain  M&S  hierarchy  provided 
additional  context  for  the  analysis.  A  top-level  review  of  the  BMD  System-Level  M&S 
fidelity,  and  the  foundations  for  radar  sensor  fidelity  followed. 

The  scope  of  this  research  included  five  large-scale.  Missile  Defense  Agency  Do¬ 
main  legacy  simulations.  The  research  methodology  supported  by  the  NPS  software  engi¬ 
neering  distance-learning  model  facilitated  the  timely  study  of  Department  primary  source 

J'('  The  Missile  Defense  Agency  received  the  Department  of  Defense  202  M&S  Award  in  the  Acquisition  category  for  the  MDA  “Enter¬ 
prise  Strategy  for  Modeling  and  Simulation”  from  Dr.  Ronald  Sega,  Director  of  Defense  Research  and  Engineering  at  a  Pentagon  cere¬ 
mony  on  September  29,  2003.  The  NPS  distance  learning  software  engineering  program  and  software  engineering  research  supported  by 
the  NPS  Software  Engineering  Department  faculty  at  Naval  Postgraduate  School,  Monterey  directly  supported  many  of  the  MDA  M&S 
program  improvements  cited  in  the  award. 
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material  for  software-intensive  legacy  simulation  systems.  This  case  study  also  employed 
selected  product  line  practice  areas  (Organization  Management,  Technical  Management, 
Software  Engineering)  as  a  tailored  framework  for  the  missile  defense  domain  analysis. 

The  analysis  identified  additional  root  causes  for  heterogeneous  anomalies  (e.g., 
substantive  interoperability  issues)  in  the  BMD  System-Level  M&S,  although  a  primary 
cause  was  not  established.  Research  indicated  that  a  number  of  factors  converged  to  create 
the  conditions  favorable  for  the  development  of  heterogeneous  system  representation 
anomalies  in  Department  simulations,  affectively  reducing  credibility  in  the  simulation  or 
federation  operations,  and  ultimately  reducing  confidence  in  the  results.  Results  from  the 
Agency  five-simulation  sample  closely  correlated  to  the  Department’s  experiences  with 
establishing  simulation  and  federation  credibility  identified  in  Chapters  II  through  Chapter 
VI. 
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X.  A  SOFTWARE  ARCHITECTURE-BASED  PRODUCT  LINE  MODEL  FOR 
SIMULATION  MODEL  REPRESENTATIONS 

A.  INTRODUCTION 

Chapter  X  introduces  the  detailed  research  design  for  the  Simulation  Software  Ar¬ 
chitecture-Based  Product  Line  Model  including  five  major  elements  supporting  the  method, 
specification,  design,  and  implementation.  The  chapter  provides  an  architectural  analysis 
of  the  Department’s  existing  de  facto  M&S  software  architecture  and  introduces  the: 

•  Simulation  Software  Architecture  fAS/fl-Bascd  Model  as  an  abstract  software  ar¬ 
chitecture-based  horizontal  foundation  supporting  multiple  viewpoints  and  views, 

•  Simulation  Software  Architectural  Framework  (SSAF),  a  second  vertical-slice267 
architectural  component  overlaying  the  SSA,  which  includes  system  and  environ¬ 
ment  components  [ABB+02,  GA03], 

•  Simulation  Product  Lines  Architecture  (SPLA)  providing  the  framework  to  man¬ 
age  the  variability268  [BBOO,  BosOO,  DMH01,  RSCOO,  MNJ+02,  CBB+03],  features 
[BosOO,  KLD02],  and  differences  between  products  comprises  the  third  element, 

•  Simulation  Software  Architecture-Based  Product  Line  Model  Domain  Metadata 
Repository  provides  the  structure  for  the  Domain  Metadata  Registry  modeled  from 
[IS079c]  to  ensure  interoperability  with  Department,  Federal  Government,  and  pri¬ 
vate  sector  metadata  registries, 

•  Architecture  Readiness  Levels  (ARL)  to  measure  future  architectural  components 
and  products  developed  from  this  methodology,  is  the  fifth  and  final  contribution 

The  SSA  and  SSAF  establish  the  architectural  foundation  for  the  SPLA,  which  sup¬ 
ports  variability  and  extensibility  of  the  architectural  construct.  We  also  adapted  the  archi¬ 
tectural  description  conceptual  framework269  from  [IEE00B]  into  the  SSA,  SSAF,  and 
SPLA  models  composing  the  Simulation  Software  Architecture-Based  Product  Line 
Model. 


The  chapter  also  suggests  a  complementary  Domain  Integrated  Product  Develop¬ 
ment  Team  (DIPDT)  approach  for  implementing  the  Simulation  Software  Architecture- 
Based  Product  Line  Model,  to  reduce  the  cycle  time  for  resolving  the  multiple  dimensions 


~67  In  UML  this  is  called  a  partition  or  a  set  of  related  classifiers  or  packages  at  the  same  level  of  abstraction  or  across  layers  in  a  layered 
architecture  [UML99], 

~68  Variability  refers  to  decisions  that  will  be  made  by  a  member  of  the  development  team  prior  to  system  deployment  [CBB+03]. 

269 

The  conceptual  framework  is  the  term  of  reference  for  the  architectural  description  and  establishes  the  terms  and  concepts  for  the 
content  and  use  of  architectural  descriptions  [lEEOOb], 
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causing  heterogeneous  system  representation  anomalies,  and  improving  federation  interop¬ 
erability. 


B.  THE  INFLUENCE  OF  SOFTWARE  ARCHITECTURE  IN  DEPARTMENT 

M&S 


The  focus  of  this  research  is  to  improve  the  quality,  accuracy,  and  consistency  of 
authoritative  representations  in  Department  M&S,  a  major  element  in  addressing  the  cur¬ 
rent  heterogeneous  system  representation  anomalies  (e.g.,  substantive  interoperability), 
which  erodes  credibility  in  the  Department  simulation  process,  and  injects  doubt  instead  of 
confidence  in  the  simulation  results  supporting  Departmental-  and  National-level  decision¬ 
makers.  This  research  indicates  that  after  fifty  years  of  development  the  Department  M&S 
domain  has  not  achieved  the  desired  level  of  credibility  and  confidence  in  simulation  model 
representations,  in  part  due  to  a  lack  of  data  sharing  and  data  standardization  problems. 
There  are  many  reasons  cited  below  for  this  assessment  including: 

•  Few  mechanisms  for  enabling  global  data  acquisition  and  interchange,  especially 
across  domain  application  areas, 

•  The  lack  of  unique  global  identifiers  for  standard  data  elements, 

•  Inadequate  documentation  of  data  element  characteristics  restricting  the  usefulness 
of  automation  to  locate,  retrieve,  and  exchange  data, 

•  A  need  for  uniform  guidance  for  identifying,  describing,  and  developing  data  ele¬ 
ments, 

•  Locating  and  retrieving  a  particular  data  element  is  difficult  or  impossible, 

•  The  absence  of  a  universal  means  for  organizing  standard  data  elements, 

•  The  lack  of  inter-organization  data  standards,  and  to  a  lesser  extent,  the  lack  of  in¬ 
tra-organization  common  data  standards, 

•  A  proliferation  of  customized  data  interchange  representations, 

•  Imprecise  data  definitions  and  descriptions,  limiting  reuse  opportunities,  or  multiple 
uses  of  the  data, 

•  A  lack  of  standard  data  elements  impeding  global  implementation  of  Electronic 
Data  Exchange  (EDI)  [IS079a], 

In  contrast  to  the  mature  computer  hardware-engineering  sector,  the  software  engi¬ 
neering  sector  of  the  computer  software  industry  is  still  developing  a  consensus  on  a  soft¬ 
ware  engineering  body  of  knowledge  [Moo99]  and  agreement  on  the  benefits  of  software 
process  improvement  and  definitions  of  software  architecture.  Although  standards  exist  for 
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three  of  the  four  components  needed  for  open  information  processing  systems  (e.g.,  hard¬ 
ware,  software,  and  communications),  standards  for  the  fourth  component  of  open  informa¬ 
tion  processing  systems,  data  specification,  is  still  under  development  by  the  international 
standards  community  and  a  relatively  new  member  of  the  Open  Systems  Interconnection 
Environment  [IS079a].  In  addition,  there  are  many  other  salient  issues  including  informa¬ 
tion  operations,  information  assurance,  and  the  security  of  personal  information  from  mis¬ 
use  for  Department  software  engineers  and  software  architects  to  consider. 

1.  The  Department’s  De  Facto  M&S  Software  Architecture 

Software  architecture,  even  if  it  is  a  de  facto  software  architecture,  strongly  influ¬ 
ences  a  system  over  its  life  cycle.  While  computer  hardware-related  architectures  sup¬ 
ported  by  Moore’s  Law  were  ascendant  over  the  past  fifty  years,  software-related  architec¬ 
tural  considerations,  if  they  existed,  were  of  secondary  importance  to  the  overall  system 
engineering  and  development  process  during  much  of  the  period.  However,  the  cost  and 
complexity  of  software-intensive  systems  changed  the  industry’s  dynamics,  as  reflected  by 
the  publication  of  software  architectural  styles  [SC96,  BK99,  HHPOO,  BosOO,  MM01, 
CBB+03,  Fra03]  descriptions  [IEEOOb],  and  patterns  [Fow97]  for  software-intensive  pro¬ 
jects  provided  new  views  and  theories  on  software  architectural  principles  to  the  software 
engineering  community.  However,  the  emerging  software  architecture  concepts  currently 
affect  little  of  the  Department’s  M&S  software  life  cycle  management  practices  for  large- 
scale,  legacy,  software-intensive  systems  and  simulation  model  representations. 

2.  A  Layered  Architecture  Model  Approach 

The  Department’s  M&S  Hierarchy  provides  the  standard  view  of  M&S  today. 
However,  the  current  M&S  Hierarchy  view  lacks  integration,  treats  abstraction  inconsis¬ 
tently,  lacks  quantitative  measurements,  and  is  more  aptly  considered  a  simulation  heuris¬ 
tic.  From  a  different  view,  the  current  de  facto  Department  M&S  software  architecture 
view  appears  closely  aligned  to  a  layered  style  .  Although  there  are  several  software 
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A  view  is  a  representation  of  a  set  of  system  elements  and  the  relationships  associated  with  them  [CBB+03],  [CBB+03]  also  viws 
each  layer  as  a  virtual  machine  with  constraints  on  the  relationships  among  the  virtual  machines. 
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architectural  styles  and  descriptions  cited  in  Chapter  VIII,  layering  supports  the  principle  of 
infonnation  hiding,  theoretically  providing  a  capability  to  change  a  lower  layer  behind  the 
interface  and  not  impact  the  layers  above  it. 

The  Department1  s  M&S  Hierarchy  continually  evolved  to  support  the  changing  re¬ 
quirements  dictated  by  new  strategies,  evolving  doctrines,  funding  justification,  changing 
organizations,  new  weapon  systems,  and  the  growing  demand  for  credible  simulations.  It 
is  clear  from  the  research  that  the  Department’s  M&S  Hierarchy  provided  a  commonly  ac¬ 
cepted  way  of  providing  a  high-level  communication  vehicle  to  convey  an  imprecise  quali¬ 
tative  definition  of  a  certain  level  of  simulation  fidelity  (e.g.,  engineering,  engagement, 
mission,  campaign).  However,  the  Departments  M&S  Hierarchy  is  not  an  architectural 
construct  and  lacks  any  clear  definition  of  the  individual  layers  or  the  interface  mechanism 
between  layers. 

In  addition,  the  hierarchical  structure  inaccurately  implies  that  the  higher  layers  of 
the  Departments  M&S  Hierarchy  build  on  the  lower  layer(s)  in  a  logical,  well-engineered 
manner.  Moreover,  since  the  Departments  M&S  Hierarchy  lacks  an  architectural  connec¬ 
tion  with  the  many  simulations  it  represents,  changes  to  either  the  simulations  or  the  hierar¬ 
chy  lack  synchronization  or  cohesion.  Lastly,  without  an  architectural  framework,  the  cur¬ 
rent  Departments  M&S  Hierarchy  appears  extremely  limited  in  its  ability  to  meet  future 
requirements  for  component-based  development  and  rapidly  composable  simulations. 

More  specifically,  the  de  facto  Department  M&S  software  architecture  view  ap¬ 
pears  to  have  two  layers, the  Conceptual  Layer  and  the  Implementation  Layer,  illustrated  in 
informal  notation  in  Figure  10-1.  The  Conceptual  Layer  theoretically  supports  and  main¬ 
tains  the  validated  CMMS  or  FDMS  conceptual  models  described  in  Chapter  V.  The  Im¬ 
plementation  Layer  is  the  software  implementation  and  deployment  of  the  M&S  system, 
theoretically  verified  against  the  validated  conceptual  model  of  the  real  world  under  study. 


Layering  reflects  a  division  of  software  into  units,  with  each  layer  representing  a  unit  [CBB+03].  [CBB+03]  also  views  each  layer  as 
a  virtual  machine  with  constraints  on  the  relationships  among  the  virtual  machines  (e.g.,  an  abstract  computing  device). 
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Implementation  Layer 

Verifies 

Conceptual  Layer 

The  Real  World  -  Layer  0 


Figure  10-1 .  Department’s  De  Facto  Two-Layer  M&S  Architecture 

In  reality,  at  Department  and  the  Agency  level,  the  theoretical  development  of  vali¬ 
dated  conceptual  models  representing  the  first  abstraction  of  the  real  world,  and  the  subse¬ 
quent  verification  of  the  simulation  software  implementation  with  the  validated  conceptual 
model  occurred  more  often  in  literature  than  in  practice  as  noted  in  the  Department’s  V&V 
experience  cited  in  Chapter  III  through  Chapter  VI;  and  the  Agency  case  study  in  Chapter 
IX.  In  addition,  the  development  of  a  validated  conceptual  model  and  verified  software 
implementation  indicates  a  high  level  of  coupling,  which  may  hinder  the  Department’s  ob¬ 
jective  for  greater  reuse  of  components  and  composable  M&S  frameworks. 
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C.  THE  HORIZONTALLY-LAYERED  SIMULATION  SOLTWARE  ARCHI¬ 
TECTURE 

1.  The  Software  Architecture  Layered  Style 

The  first  element  of  the  Simulation  Software  Architecture-Based  Product  Line 
Model  is  the  Simulation  Software  Architecture  (SSA).  The  proposed  simulation  software 
architecture  style  is  the  layered  style.  The  layered  view  of  the  Department’s  simulation 
software  architecture,  graphically  presented  as  a  layer  diagram,  segments  software  into 
each  layer  with  constraints,  connected  by  a  public  interface.  Properly  developed  layered- 
style  diagrams  support  the  development  of  software-intensive  simulation  systems  featuring 
portability,  interoperability,  and  modifiability  facilitating  reuse,  component-based  devel¬ 
opment,  and  future  composability  research  initiatives.  The  layered  diagram  sees  common 
use,  although  some  software  architects  employ  poorly  constructed  layered  diagrams,  often 
using  facilities  of  a  higher  layer  without  restrictions. 

Properly  constructed  layered  architecture  diagrams  share  a  basic  quality  in  which 
the  layers  interact  according  to  a  strict  ordering  relationship.  Layered  diagrams  employ  an 
allowed  to  use  relationship.  In  the  proposed  simulation  software  architecture  (SSA),  the 
Referent  Layer  is  beneath  the  Conceptual  Layer  and  provides  the  views  and  viewpoints  into 
the  real  world  (e.g.,  Layer  0).  The  implementation  of  the  Conceptual  Layer  may  use  all  of 
the  public  facilities  provided  by  the  Referent  Layer  through  the  interface.  The  Conceptual 
Layer  is  beneath  the  Component  Layer.  The  implementation  of  the  Component  Layer  may 
use  all  of  the  public  facilities  provided  by  the  Conceptual  Layer  through  the  interface.  The 
Component  Layer  is  beneath  the  Implementation  Layer.  The  Implementation  Layer  may 
use  all  of  the  public  facilities  provided  by  the  Component  Layer  through  the  interface. 

We  reviewed  several  other  architectural  styles  as  candidates  for  the  SSA,  including 
pipes  and  filters  [SC96],  layers  [BBG+00],  blackboards  [BosOO],  object-orientation,  dis¬ 
tributed  and  systems,  component  and  connector,  interactive  systems  and  several  other 
styles  described  in  Chapter  VIII.  [BosOO]  describes  this  activity  as  imposing  an  architec- 
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“  The  allowed  to  use  is  a  variant  of  the  depends-on  relation,  where  if  Pi  uses  P2.  Prs  correctness  depends  on  the  correct  implementation 
of  P2  [CBB+03], 
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tural  style  on  the  software  architecture,  requiring  careful  consideration  in  the  process  and 
the  impact  of  a  complete  reorganization  of  the  software  architecture.  The  primary  goals  of 
the  SSA  are  to  provide  an  architectural  framework  to  support  constantly  changing  require¬ 
ments  and  secondly,  reduce  simulation  software  complexity.  Quality  attributes  support 
each  SSA  layer.  Figure  10-2  illustrates  the  proposed  SSA  model. 

In  addition  the  review  encompassed  the  decomposition  style,  uses  style  (e.g.,  de- 
pends-on  relationship),  generalization  style  (e.g.,  is -a  relationship),  and  the  layered  style 
[CBB+03].  The  selected  stack  layer  model  for  the  SSA  uses  geometric  adjacency  to  repre¬ 
sent  the  allowed-to-use  protocol,  as  opposed  to  symbols  (e.g.,  arrows,  lines,  diamonds). 
Other  infonnal  layered  notations  evaluated  for  the  SSA  layered  style  included  segmented 
layers,  rings,  three-dimensional  models,  and  layers  with  sidecars  [CBB+03],  with  which  we 
compared  and  contrasted  the  architectural  attributes  against  the  informal  stack  notation 
which  represents  the  layers  as  a  stack  of  rectangles. 

The  SSA  restricts  upward  communication  by  a  lower  layer  of  the  architecture  with 
the  facilities  of  higher  layers  to  ensure  the  validity  of  the  desired  architectural  attributes  and 
to  retain  the  required  properties  to  support  the  V&V  process.  Each  horizontal  slice  or  layer 
of  the  stack  supports  a  vertical  slice  or  abstraction  of  the  product  line,  discussed  later  in  the 
chapter,  and  no  two  layers  contain  the  same  product  line  abstraction.  The  thick  horizontal 
lines  between  the  layers  indicate  the  constraints  of  interlayer  communication  to  the  inter¬ 
face  protocols  at  the  layer  interface,  with  no  other  communications  to  the  internal  facilities 
of  any  other  layer. 
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Implementation  Layer 

Component  Layer 

Conceptual  Layer 

Referent  Layer 

The  Real  World  -  Layer  0 

Figure  10-2.  Simulation  Software  Architecture  (SSA)  Model 

2.  Viewpoints  and  Views 

A  viewpoint  provides  the  pattern  or  framework,  based  on  the  stakeholders’  intended 
use  of  the  system,  for  specifying  the  view  design  and  development.  Viewpoints  selected 
for  the  simulation  software  architecture  description  (SimSAD)  may  originate  from  the  de¬ 
velopment  effort  or  defined  elsewhere  and  reused  in  the  SimSAD,  tenned  a  library  view  by 
[IEE00B].  Stakeholders  select  one  or  more  viewpoints  for  the  architectural  description  of 
the  system  supporting  the  intended  use  of  the  simulation.  A  viewpoint  sets  the  modeling 
methods,  procedures  and  analysis  techniques  for  analyzing,  creating,  and  displaying  a  view 
representation.  The  minimum  specification  for  viewpoint  metadata  maintained  in  the 
Simulation  Software  Architecture-Based  Product  Line  Model  Domain  Metadata  Repository 
includes: 

•  The  viewpoint  name, 

•  List  of  stakeholders  supported  by  the  viewpoint, 
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•  The  intended  use,  mission,  and  concerns  addressed  by  the  viewpoint, 

•  The  language,  modeling  techniques,  analysis  methods  used  to  develop  a 
view  based  on  the  viewpoint, 

•  The  source  for  the  viewpoint  or  library  viewpoint  including  data  sources 
for  validation, 

•  Formal  or  informal  consistency  and  completeness  tests  applied  to  the 
models  used  to  develop  a  view, 

•  Evaluation  or  analysis  techniques  to  be  applied  to  the  models, 

•  Patterns,  heuristics,  and  other  guidelines  supporting  the  development  of 
a  view  [IEEOOb]. 

We  define  a  view  as  a  representation  of  a  whole  system  or  set  of  system  elements 
from  the  perspective  of  a  related  set  of  concerns  and  the  relationships  associated  with  them. 
The  view  addresses  stakeholders’  mission  or  intended  use  of  the  system  in  the  simulation 
or  federation.  A  view  may  also  include  one  or  more  architectural  models,  which  may  sup¬ 
port  one  or  more  views,  derived  from  another  associated  viewpoint  architecture. 

Viewpoints  and  views  are  major  contributors  to  this  dissertation.  Take  the  example 
of  a  house.  The  electrician,  carpenter,  plumber,  bricklayer,  HVAC  (e.g.,  heating,  ventila¬ 
tion,  and  cooling)  contractor,  cement  contactor,  and  landscape  specialist  all  have  different 
viewpoints  of  the  same  house.  All  these  viewpoints  are  correct  since  they  support  the  mis¬ 
sion  or  intended  use  for  the  rules,  procedures,  and  guidelines  (e.g.,  appropriate  building 
codes)  the  different  craftsmen  must  follow  to  develop  the  different  views  of  the  architect, 
surveyor,  builder,  and  owner.  Different  views  are  also  important,  since  the  builder  wants 
the  potential  owner  to  buy  the  house,  and  later  on  the  homeowner  hopes  the  tax  appraiser 
takes  a  lower-valued  view  of  the  property,  while  potential  new  buyers  take  a  higher-valued 
view. 


3.  The  Referent  Layer 

The  Referent  Layer  is  the  architectural  framework  of  the  SSA  supporting  view¬ 
points.  Previous  authors  cited  the  need  for  a  referent  as  a  commonly  understood  standard 
[Gro99,  RGHOO],  or  a  properly  developed  conceptual  model  supporting  fidelity  [GFT98, 
PacOlb]  and  validation  [PacOlb].  We  view  system  referents  as  software  architecture  con¬ 
structs,  and  components  of  the  SimSAD,  populating  the  first  layer  of  the  SSA,  the  Referent 
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layer.  The  single  SSA  Referent  Layer  in  Figure  10-3  replaces  all  levels  of  the  Depart¬ 
ments  M&S  Hierarchy  discussed  in  Chapter  II  (see  Figure  2-2)  and  employs  viewpoints  of 
the  real  world  as  portals  into  the  architectural  framework.  Viewpoints  are  new  to  the  De¬ 
partment’s  simulation  software  architecture  design  process.  Viewpoints  represent  a  one-to- 
one  mapping  from  the  real  world  into  the  architectural  sink,  the  system  referent,  maintained 
in  the  referent  Layer.  Referents  support  one  or  more  real-world  viewpoints. 


Figure  10-3.  SSA  Referent  Layer  Model 

The  Referent  Layer  performs  the  transaction  manager  role  for  the  SSA.  The  Refer¬ 
ent  Layer  continually  adds  new  viewpoints  to  the  referent,  while  existing  viewpoints 
change,  evolve,  and  merge  to  new  meet  requirements,  until  archived  pending  future  use. 
The  Referent  Layer  provides  the  first  interface  to  the  services  provided  by  the  Simulation 
Software  Architecture-Based  Product  Line  Model  Metadata  Repository.  This  process  ap¬ 
plies  to  all  data  sources  including  new  system  requirements  and  the  core  asset  development 
methods  discussed  later  in  the  chapter,  including  mining  and  reverse  engineering. 
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4. 


The  Conceptual  Layer 


The  Conceptual  Layer  in  Figure  10-4  houses  the  conceptual  views  of  the  systems. 
Where  the  Referent  Layer  provided  the  transaction  manager  role  for  the  SSA,  the  Concep¬ 
tual  Layer  provides  the  facilities  for  conceptual  composition.  The  major  component  of  the 
Conceptual  Layer  is  the  Simulation  System  Architectural  Description  (SimSAD).  Each 
SimSAD  comprises  a  specific  system,  which  may  include  product  lines,  product  families, 
complete  enterprises,  aggregated  entities,  subsystems,  components,  systems,  and  systems 
of  systems,  and  their  interfaces  inhabiting  an  environment. 


Figure  10-4.  SSA  Conceptual  Layer  Model 

The  Conceptual  Layer  provides  facilities  to  “pull”  viewpoints  and  referents  from 
the  Simulation  Software  Architecture-Based  Product  Line  Model  Metadata  Repository  pro¬ 
vided  by  the  Referent  Layer  to  support  the  development  of  views  for  the  SimSAD  and  con¬ 
ceptual  model  construction.  Conceptual  views  support  the  development  of  specific  concep¬ 
tual  models  designated  for  an  intended  use  or  mission;  or  common-use  conceptual  models; 
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or  conceptual  models  specifically  designed  for  large-scale  reuse.  The  Simulation  Software 
Architecture-Based  Product  Line  Model  Metadata  Repository  provides  configuration  con¬ 
trol  services  for  all  views  in  the  Conceptual  Layer,  and  provides  the  interface  control  with 
the  next  higher  layer,  the  Component  Layer. 

5.  The  Component  Layer 

The  Component  Layer  in  Figure  10-5  houses  and  manages  the  component  views. 
Where  the  Conceptual  Layer  provided  the  facilities  for  architectural  conceptual  composi¬ 
tion,  the  Component  Layer  provides  facilities  to  “pull”  views  from  the  Simulation  Software 
Architecture-Based  Product  Line  Model  Metadata  Repository  provided  by  the  Conceptual 
Layer  to  support  the  development  of  design  strategies  for  component  models. 


Referent  Layer 


Figure  10-5.  SSA  Component  Layer  Model 

Conceptual  views  support  the  development  of  patterns  of  interaction  for  specific  compo¬ 
nent  models  designated  for  an  intended  use  or  mission;  or  common-use  component  models; 
or  component  models  specifically  designed  for  large-scale  reuse. 
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The  Component  Layer  provides  the  architectural  foundation  for  future  composable 
item  development  by  providing  standards  and  conventions  for  quality  attributes 
[WBL+01],  composition  rules,  and  visibility  of  assets.  Components  were  discussed  in 
Chapter  VIII.  The  Simulation  Software  Architecture-Based  Product  Line  Model  Metadata 
Repository  provides  configuration  control  services  for  all  SimSAD  components  in  the 
Component  Layer,  and  provides  the  interface  control  with  the  next  higher  layer,  the  Im¬ 
plementation  Layer,  which  “pulls”  component  views  as  required.  The  development  of  ap¬ 
plication  components  is  beyond  the  scope  of  this  research. 

6.  The  Implementation  Layer 

The  Implementation  Layer  in  Figure  10-6  houses  and  maintains  the  complete  Sim¬ 
SAD  system  view,  and  provides  the  visibility  of  the  component  views  available  to  imple¬ 
mentation  layer  developers.  Where  the  Conceptual  Layer  provided  the  facilities  for  con¬ 
ceptual  composition,  the  Implementation  Layer  provides  facilities  to  “pull”  component 
views  from  the  Simulation  Software  Architecture-Based  Product  Line  Model  Metadata  Re¬ 
pository  provided  by  the  Component  Layer  to  support  the  development  of  system  model 
views.  Component  views  support  the  development  of  specific  system  models  designated 
for  an  intended  use  or  mission;  common-use  system  models;  or  system  models  specifically 
designed  for  large-scale  reuse. 

The  Implementation  Layer  provides  the  architectural  foundation  for  future  com¬ 
posable  system  development.  The  Simulation  Software  Architecture-Based  Product  Line 
Model  Metadata  Repository  provides  configuration  control  services  for  all  systems  in  the 
Implementation  Layer  and  provides  the  interface  control  with  the  next  lower  layer,  the 
Component  Layer,  “pulling”  component  views  when  required.  The  development  of  com¬ 
posable  application  systems  is  beyond  the  scope  of  this  research. 
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Conceptual  Layer 


Figure  10-6.  SSA  Implementation  Layer  Model 

7.  The  Simulation  Software  Architecture  Framework  (SSAF) 

The  second  element  of  the  Simulation  Software  Architecture-Based  Product  Line 
Model  is  the  Simulation  Software  Architecture  Framework  (SSAF),  which  provides  the 
system  environment  components.  The  SSAF  in  Figure  10-7  adds  the  vertical  dimension  of 
the  system  environment  to  the  horizontal  architectural  layer  presented  in  the  SSA.  The  re¬ 
quirement  for  a  vertical  dimension  is  an  unusual  requirement  since  many  Department  soft¬ 
ware  systems  developed  in  the  past  have  evolved  in  a  vertical  manner,  often  referred  to  as 
“stove-piped”  systems  (e.g.,  personnel,  logistics,  medical,  command  and  control),  and 
lacked  horizontal  integration  and  interoperability.  However,  current  M&S  Hierarchy  mod¬ 
els  lacks  structure  to  vertically  integrate  the  system  model  representations  in  a  consistent 
fashion. 

Although  the  Department’  simulation  development  history  experienced  a  similar 
pattern,  architecturally  it  developed  a  different  growth  pattern.  The  simulation  framework 
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and  its  internal  models  presented  a  de  facto  self-contained  horizontal  architecture  layer, 
within  the  simulation  framework  (e.g.,  engine,  scenarios)  providing  the  horizontal  integra¬ 
tion  and  interoperability.  However,  the  Department’s  M&S  Hierarchy  provided  few  verti¬ 
cal  integration  enablers  supporting  a  cohesive  process  for  managing  quality  attributes  for 
the  same  system  at  different  levels  of  the  M&S  Hierarchy.  This  systemic  shortcoming  ad¬ 
versely  affected  the  Department’s  MS  credibility.  The  SSAF  presents  a  method  to  improve 
the  Department’s  vertical  integration  of  M&S.  The  system  component  of  the  SSAF  sup- 
ports  the  SimSAD.  The  system  environment  ,  includes  the  physical  world  and  the  exter¬ 
nal  objects,  conditions,  or  processes  influencing  the  behavior  of  the  system  in  past,  present, 
and  future  dimensions,  real  or  imagined,  but  clearly  defined  and  documented  in  the  Sim¬ 
SAD. 

A  system  has  one  or  more  stakeholders,  with  interests,  intended  uses  and  con- 
cerns"  .  SimSAD  stakeholders  include  simulation  developers,  architects,  simulation  spon¬ 
sors,  V&V  agents,  accreditation  agents,  and  ultimately  Agency-,  Department-,  and  Na¬ 
tional-level  decision-makers.  A  system  design  also  incorporates  the  intended  use,  or  ful- 
fillment  of  a  mission"  in  the  system’s  environment.  We  use  the  terms  mission  and  in¬ 
tended  use  interchangeably.  The  architectural  description  component  of  the  SimSAD  in¬ 
cludes  the  compilation  of  products  to  document  architectures  [IEEOOb]. 
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The  system's  environment  or  context  may  influence  the  system’s  setting,  including  circumstances  of  operational,  developmental, 
political  and  other  influences  affecting  the  system.  The  environment  may  include  other  systems’  direct  or  indirect  interaction  with  the 
system  of  interest  through  defined  interfaces,  and  also  defines  the  boundaries  and  scope  for  the  system  of  interest  {IEEOO], 
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Stakeholder  Concerns  include  the  system’s  development,  operation,  or  any  aspect  of  the  system  deemed  critical  by  the  stakeholder, 

including  quality  attributes  detailed  in  Chapter  VI. 
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A  mission  involves  the  operation  or  use  of  system  by  its  stakeholders  to  meet  objectives  developed  for  the  systems  intended  use 
[IEEOOb], 
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Layer  0  -  The  Real  World 


Figure  10-7.  Simulation  Software  Architecture  Framework  (SSAF) 

Recall  that  all  systems  have  a  defined  or  de  facto  architecture.  The  SimSAD  de¬ 
scribes  the  conceptual  architecture  of  a  system  and  supports  the  follow-on  development  of 
product  line  descriptions  for  particular  discrete  instantiations  of  that  specific  architecture. 
The  SimSAD  maintains  one  or  more  constituent  views  and  viewpoints.  In  addition  to  the 
selected  views  and  viewtypes  the  SimSAD  maintains  other  information  and  documentation 
supporting  verification,  validation,  and  accreditation  processes,  including  a  system  over¬ 
view,  system  context,  product  family  relationships,  URLs  for  system  validation  data,  and 
maintenance  information.  [IEEOOb]  provides  a  concise  listing  of  uses  of  architectural  de¬ 
scriptions,  architectural  description  practices,  and  recommended  documentation  supporting 
a  SimSAD,  including  the  specification  for  a  group  of  systems  sharing  a  common  set  of  fea¬ 
tures  (e.g.,  product  lines).  The  SimSAD  also  maintains  records  and  analysis  data  of  known 
architectural  inconsistencies. 
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8. 


Software  Architecture  Reconstruction 


Software  architecture  reconstruction  provides  a  mechanism  for  the  effective  reuse 
of  software  assets  [KC97,  CW98,  ObrOla,  ObrOlb,  KovOl,  OSV02].  Architecture  reuse  is 
the  cornerstone  practice  of  product  line  development,  supported  by  three  sources  for  archi¬ 
tecture  recovery: 

•  Source  code,  which  is  authoritative,  but  does  not  hold  all  the  information, 

•  Documentation,  which  is  normally  undependable,  incomplete,  or  outdated, 

•  Human  experts,  who  may  be  dependable,  but  biased  [CW98]. 

The  SEI  developed  the  Options  Analysis  for  Reengineering  (OAR)  methodology 
[BSW+99,  BSOO,  CN02]  for  making  technical,  organizational,  and  programmatic  reengi¬ 
neering  decisions.  The  OAR  methodology: 

•  Combines  architectural  and  reengineering  views  with  defined  mappings  between 
the  views, 

•  Classifies  and  stratifies  reengineering  options/app roaches  into  distinct  layers  with  a 
mapping  between  layers, 

•  Provides  key  information  for  making  informed  choices  about  the  time  and  criteria 
for  each  option/approach, 

•  Codifies  technical  and  non- technical  risks  for  each  option/approach, 

•  Relates  organizational  and  programmatic  factors  to  support  a  unified  reengineering 
option/approach  [BSW+99]. 

[BSW+99]  indicate  the  reengineering  process  includes  three  basic  steps  to  evolve 
an  existing  legacy  into  a  new  system  in  a  disciplined  evolutionary  process  employing  the 
Horseshoe  Model,  in  Figure  10-8,  which  combines  reengineering  and  architectural  views  of 
software  analysis  and  evolution  including: 

•  Analysis  of  the  existing  system,  and  extracting  artifacts  from  source  code,  the  archi¬ 
tecture  recovery  /  conformance  process, 

•  The  architectural  transformation  of  the  legacy  systems  logical  descriptions  into  new 
improved  descriptions, 

•  Architecture-Based  Development  (ABD)  of  the  new  system  based  on  the  new 
logical  descriptions  to  instantiate  the  desired  architecture  [BSW+99]. 

The  Horseshoe  Model  [BSW+99]  shown  in  Figure  10-8  includes  three  levels. 

These  levels  according  to  [BSW+99]  consist  of  a  code-structured  representation,  which  in¬ 
cludes  parsing  and  analysis  of  the  source  code,  abstract  syntax  trees  (ASTs),  and  flow 
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graphs  at  the  first  level,  and  a  functional  level  representation  describing  the  relationships  of 
functions,  data  and  files  at  the  second  level.  At  the  third  level,  the  Horseshoe  Model  illus¬ 
trates  the  concept  level  representing  combined  function  and  code  level  artifacts  used  as  the 
base  of  new  architectural  components  or  concepts. 
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Figure  10-8. 


The  Horseshoe  Model  Underlying  OAR  (From  [BSW+99]) 


The  source-code  level  transformations  may  employ  string  matching  and  replace¬ 
ment  techniques,  or  code-structure  transformations  based  on  syntax  tree-based  transforma¬ 
tions.  Functional  level  transformations  go  beyond  code-level  changes  and  are  concerned 
primarily  with  the  reworking  of  the  functionality,  which  may  include  changing  from  a 
functional  design  to  an  object-oriented  design.  The  final  transformation  process,  at  the  ar¬ 
chitectural  level,  involves  changes  to  the  basic  building  blocks  including  the  typed  of  com¬ 
ponents,  patterns  of  interaction,  control  mechanism,  functional  allocation,  and  data,  and 
normally  requires  the  greatest  amount  of  time  and  resources  according  [BSW+99]. 

Architecture  recovery  and  reconstruction  is  an  iterative  and  interactive  labor- 
intensive  effort,  dependent  upon  the  level  of  specification,  documentation,  dissemination 
and  control  of  the  legacy  architecture  artifacts  [KOVOl,  OSV02].  Architecture  recovery 
methods  vary  from  entirely  manual  reconstruction  processes  to  tool-supported  manual  re¬ 
construction  and  semi-autonomous  reconstruction  techniques,  and  may  include  data  min- 
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ing276  and  employment  of  architecture  description  languages  [OSV02].  Pattern  analysis 
supports  architecture  recovery  and  reconstruction  efforts  [OSV02].  [Fow97]  addresses  two 
categories  of  patterns: 

•  Analysis  patterns  are  groups  of  concepts  that  represent  a  common  construction  in 
business  modeling,  relevant  to  one  or  many  domains, 

•  Supporting  patterns  are  patterns  in  their  own  right  and  valuable  on  their  own,  al¬ 
though  the  primary  purpose  supports  the  use  of  analysis  patterns,  making  them  real 
[Fow97]. 

Manual  mining  activities  recover  source  infonnation  for  architecture  activities  sup¬ 
porting  product  line  development  from  the  source  code,  documentation,  and  human  ex- 
perts,  since  automated  tools'  are  currently  limited.  [KOVOl,  OSV02A]  suggest  the  use 
and  synthesis  of  existing  several  tools  and  techniques  into  a  “workbench”  support  envi¬ 
ronment  for  software  analysts  reconstructing  architectures.  A  significant  number  of  tools 
support  view  extraction,  and  several  tools  support  view  fusing  and  reconstruction. 

[HBL97,  KC97,  ObrOla,  ObrOlb,  BBC+02]  support  architecture  reconstruction 
with  automated  tools;  however,  [OSV02]  note  that  tool  use  is  limited  by  a  system  using 
several  languages,  or  on  cases  where  the  binary  code  is  available,  but  not  the  source  code. 
The  possible  lack  of  source  code  is  a  real  possibility  in  reconstruction  efforts  with  commer¬ 
cial  components  and  the  vendors  decline  to  provide  the  source  code  [OSV02]. 

D.  THE  VERTICALLY-SLICED  SOFTWARE  PRODUCT  LINE  ARCHITEC¬ 
TURE 

[BosOO]  suggests  three  purposes  for  architecture:  as  an  individual  software  system, 
as  product-line  architecture278,  or  as  a  standard  architecture  for  component  development. 
Research  indicates  that  component-based  technologies  remain  immature,  suggesting  that  an 
organizationally  sponsored  (e.g.,  domain)  product  line-based  approach  to  implementing 
simulation  software  architectures,  may  serve  as  a  predecessor  technology  to  a  follow-on 
component-based  development  environment. 


~76  Mining  includes  resurrecting  and  rehabilitating  a  piece  of  an  existing  software  system  to  serve  in  a  new  system  for  which  it  was  not 

originally  intended  [Nor03], 

277 

One  existing  tool  is  Dali  is  an  SEI  tool  for  extracting  architectural  information  from  an  implemented  system  [CW98] 

~78  [BosOO]  defines  the  use  of  a  product-line  architecture  as  the  common  architecture  for  a  set  of  related  products  or  systems  developed 
by  an  organization. 
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[BosOO]  also  identified  two  factors  for  the  approach  to  initiating  a  product  line:  evo¬ 
lutionary  or  revolutionary;  and  suggested  two  market-based  decisions  to  develop  a  new 
product  line  or  apply  a  product  line  methodology  to  a  legacy  product.  Risk  plays  a  major 
role  in  the  decision  process,  especially  in  mission-critical,  high-risk  Department  domains. 
Employing  an  evolutionary  product  line  approach  for  large-scale,  software-intensive  legacy 
simulation  systems  has  a  potential  in  the  Department’s  M&S  domain  to: 

•  Minimize  investment  risk  and  risks  to  continuity  of  operations, 

•  Support  the  development  of  authoritative  representations, 

•  Reduce  federation  issues  caused  by  substantive  interoperability  problems, 

•  Identify  the  level  of  quality  in  the  component  product,  and  not  just  the  level  of  ma¬ 
turity  in  the  processes  supporting  development, 

•  Support  improved  VV&A  practices, 

•  Provide  metric/measured  credibility  of  Department  M&S  supporting  improved  con¬ 
fidence  levels, 

•  Initiate  the  transition  of  Department  M&S  to  a  supportable  Department  software  ar¬ 
chitecture, 

•  Capitalize  on  the  current  core  asset  investment, 

•  Improve  component  syntactical  /  semantic  inconsistencies  between  Department 
domains, 

•  Provide  a  foundation  for  an  integrated  data  and  object  model  ontology, 

•  Provide  a  common  open  source  for  validated  algorithms, 

•  Reduce  overall  life  cycle  support  costs. 

It  is  possible  to  implement  a  software  product  line  architecture  shown  in  Figure  10- 
9  at  various  defined  levels — for  example:  enterprise,  domain,  functional,  system,  subsys¬ 
tem,  or  component  level.  A  key  step  in  the  systems  and  software  engineering  process  is  the 
iterative  allocation  of  system  requirements  to  hardware  and  software.  This  is  also  true  for 
software-intensive  product  lines  since  the  system  may  include  hardware  (e.g.,  hardware  in 
the  loop)  or  live  simulation  entities. 
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Layer  0  -  The  Real  World 


Figure  10-9.  Software  Product  Line  Architecture  (SPLA) 


The  domain-level  system  and  software  engineering  process  defines  the  software 

279  280 

product  line  standard  and  product  specification  .  These  documents  provide  the  alio- 
cated  level  of  product  line  system  abstraction"  and  data  abstraction"  ,  including  accu- 
racy  ,  precision"  and  quality  factors  or  quality  attributes  ,  allocated  as  requirements 
to  systems,  subsystems,  sets,  groups,  units,  components,  assemblies,  subassembly,  and  fi- 
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A  standard  that  defines  what  constitutes  completeness  and  acceptability  of  items  that  are  used  or  produced,  formally  or  informally, 
during  the  software  engineering  process  [IEE90], 

~80  The  product  specification  is  a  critical  component  of  product  lines,  and  is  essential  to  support  variability.  (1)  It  specifies  the  design 
that  production  copies  of  a  system  or  component  must  implement,  or  (2)  a  document  that  describes  the  characteristics  of  a  planned  or 
existing  product  for  consideration  by  potential  customers  or  users  [IEE90], 

~81  Abstraction  is  (1)  a  view  of  the  object  that  focuses  on  the  information  relevant  to  a  particular  puipose  and  ignores  the  remainder  of 
the  information.  (2)  The  process  of  formulating  a  view  as  in  (1)  [IEE90], 

~8~  The  process  of  extracting  the  essential  characteristics  of  data  by  defining  data  types  and  their  associated  functional  characteristics  and 
disregarding  representational  detail  [IEE90], 

Accuracy  in  product  lines  is  either  (1)  a  qualitative  assessment  of  correctness,  or  freedom  from  error,  or  (2)  a  quantitative  measure  on 

the  magnitude  of  error  [IEE90], 

284 

Precision  is  the  degree  of  exactness  or  discrimination  with  a  quantity  is  stated  [1EE90], 

~85  A  quality  factor  is  a  management-oriented  attribute  of  software  that  contributes  to  its  quality  [IEE98d]. 

~86  A  characteristic  of  software,  or  a  generic  term  applying  to  quality  factors,  quality  sub  factors,  or  metric  value  [!EE98d],  See  Chapter 
VI  for  a  more  detailed  discussion  on  quality. 
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nally  the  part  level  composing  the  product  line.  Product  line  quality  attributes  establish  the 
foundation  for  product  line  software  quality  metrics  and  product  metrics  ,  essential  to 
maintaining  control  of  product  line  variability. 

The  software  product  line  architecture  also  supports  a  methodology  independent 
classification  scheme289,  classified  scheme  items290,  and  classified  components291  for  sev- 
eral  components  of  data  elements  including  object  classes'  ,  properties"  ,  representa¬ 
tions294,  value  domains295,  data  element  concepts,  as  well  as  actual  data  elements,  including 
keywords"  ,  thesaurus  terms"  ,  taxonomy  ,  and  ontology"  taxa  illustrated  in  Figure 
10-10.  Classification  attributes301  associate  various  classification  schemes  with  selected 
components  of  data  elements  [IS079b]  including: 

•  Classified  component  ID, 

•  Classified  component  name. 

•  Classification  scheme  type, 

•  Classification  scheme  name, 

•  Classification  scheme  version, 

•  Classification  scheme  item  type, 

•  Classification  scheme  item  value  [IS079b]. 


~87  A  function  whose  inputs  are  software  data  and  whose  output  is  a  single  numerical  value  that  can  be  interpreted  as  the  degree  to  which 
the  software  possesses  a  given  attribute  that  affects  its  quality  [!EE98d], 

~8S  A  metric  used  to  measure  the  characteristics  of  any  intermediate  or  final  product  of  the  software  development  process  [IEE98d]. 

289 

The  arrangement  or  division  of  objects  into  groups  based  on  characteristics  that  the  objects  have  in  common  (e.g.,  origin,  composi¬ 
tion,  structure,  application,  and  function)  [IS079bj. 

290 

Classified  scheme  items  are  Components  of  content  in  a  classification  scheme  [IS079b], 

291 

A  component  of  a  data  element  that  may  be  classified  in  one  or  more  classification  schemes  [IS079b], 

292 

“  An  object  class  is  a  set  of  ideas,  abstractions,  or  things  in  the  real  world  that  can  be  identified  with  explicit  boundaries  and  meaning 

and  whose  properties  and  behavior  follow  the  same  rules  [!EE97a], 
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A  property  is  a  peculiarity  common  to  all  members  of  an  object  class  [IS079a], 

294 

A  representation  describes  how  the  data  are  represented,  and  if  necessary,  a  unit  of  measure  or  a  character  set  [!S079a], 

295 

A  value  (e.g.,  data  value)  domain  is  a  set  of  permissible  values,  which  may  be  enumerated  or  expressed  by  a  description.  The  value 
domain  provides  representation,  but  does  not  include  the  data  element  concept  the  values  may  be  associated  with  nor  what  the  value 

means  [IS079c], 
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Keywords  are  basic  attributes  applied  to  object  classes,  properties,  representations,  and  data  elements  and  include  the  definition, 
obligation,  data  type,  and  comment  [IS079b], 
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Thesaurus  terms  can  be  associated  with  data  elements  and  data  element  concepts  [lS079b], 

298 

Taxonomy  is  a  hierarchical  organization  of  concepts  (e.g.,  taxa)  based  on  generalization/specialization  and  the  mathematical  notions 

of  sets,  subsets,  and  set  membership  [lS079b], 

299 

Ontology  is  a  network  organization  of  taxa  meant  to  provide  a  model  of  some  portion  of  the  world,  and  consists  of  theories  about  the 
sorts  of  objects,  properties  of  objects,  and  relations  among  objects  that  are  possible  in  that  portion  of  the  world  [IS079b], 

300  The  taxa  in  taxonomies  and  ontologies  may  be  related  to  classified  data  registration  components  including  object  class,  property, 

registration  class,  and  data  element  concept  [lS079b]. 

301 

Classification  attribute  descriptions  include  name,  definition,  obligation,  condition,  data  type,  and  comments  [IS079b], 
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CCOOOI  =  object  dais  (e.g. ,  fruit  fly)  = Drosophila  -  taxon  lent) 


Information 
System  ( biology ) 


Figure  10-10.  Software  Architecture  Classification  (From  [IS079c]) 

A  software  product  line  approach  may  be  employed  at  one  or  more  levels,  depend¬ 
ing  on  many  factors  such  as  the  application  domain,  degree  of  commonality  and  feature 
variability  with  other  systems  /  sub-systems  /  components,  and  availability  of  candidate 
(reusable)  software  assets.  To  illustrate  these  engineering  concepts,  consider  a  missile  sys¬ 
tem  in  a  simulation  as  an  integrated  system  of  subsystems  and  components  performing  cer¬ 
tain  functions  for  the  missile,  such  as  navigation.  There  are  many  options,  including  the 
establishment  of  a  product  line  program  for  a  computational  system  used  across  a  family  of 
missiles,  or  a  product  line  program  established  for  the  navigation  subsystems  supplied  to 
many  missiles. 

The  best  leverage  of  simulation  software  architecture-based  product  lines  occurs 
when  they  share  a  common  architecture  employing  quality  components  for  consistent 
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products.  Each  level  of  the  M&S  software  architecture-based  product  line  model  would 
have  discrete  quantifiable  performance  and  quality  attributes. 

1.  Core  Architecture  Asset  Development 

a.  Mining  for  Product  Line  Core  Architecture  Assets 

Systemic  software  reuse  focuses  on  large-grained  reusable  assets  such  as 
software  architectures,  processes,  documentation,  test  cases,  and  components,  versus  the 
previous  efforts  in  software  reuse  that  revolved  around  small-grained  assets  of  software 
code,  which  experienced  only  modest  gains  [BFG+00,  CN02].  The  system  architecture, 
documentation,  and  components  are  central  core  assets  used  to  evolve  and  build  the  soft¬ 
ware  product  line. 

[BSOOO,  BOSOO,  CN02]  propose  existing  assets  offers  an  organization  the 
potential  to  leverage  all,  or  part,  of  its  cumulative  system  investments,  and  represents  a 
critical  practice  area  supporting  a  software  product  line  initiative.  However,  [BSOOO, 
CN02]  cite  significant  risks  achieving  success  developing  software  product  lines.  [BSOOO] 
attributes  the  high  risks  to  the  lack  of  documentation,  the  poorly  maintained  state  of  many 
existing  systems,  and  the  fact  that  many  systems,  initially  developed  for  different  purposes, 
lack  support  for  current  software  engineering  approaches.  Research  indicates  this  is  gener¬ 
ally  true  in  the  Department’s  M&S  domain.  [BSOOO,  CN02]  identify  four  basic  steps  to 
successfully  mine  assets:  1)  preliminary  information  gathering,  2)  making  decisions  on 
whether  to  mine  assets  and  which  type  of  overall  strategy  to  use,  3)  obtaining  detailed 
technical  understanding  of  existing  software  assets,  and  4)  rehabilitation  of  assets. 

In  mining  legacy  assets,  [BSOO,  BOSOO]  suggest  the  focus  should  be  on 
mining  specific  legacy  software  and  evaluating  adaptation  techniques  for  product  line  use, 
as  opposed  to  either  code  level  transformation  or  reengineering  the  system  entirely.  [BSOO] 
suggests  the  focus  on  architectural  compatibility  and  interfaces  involves  the  identification 
of  large-grained  legacy  system  functionality,  and  mining  black-box  software  elements 
available  for  adaptation  or  wrapping  to  serve  as  core  assets,  or  support  architectural  extrac¬ 
tion  and  reconstruction.  Tradeoffs  happen  continuously  in  the  early  mining  phases  to  re- 
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solve  technical  challenges,  while  constantly  evaluating  legacy  software  assets  for  mining 
potential,  and  to  detennine  the  practicality  and  cost  effectiveness  of  the  mining  operation. 

b.  Reverse  Engineering  for  Product  Line  Core  Architecture  Assets 

[Til98]  introduces  a  reverse-engineering302  environment  framework  to  im- 
prove  program  understanding  based  on  a  descriptive  model  with  categories  of  support 
mechanism  features  established  on  a  foundation  of  attributes.  [Til98]  identifies  three 
groupings  of  support  mechanism  category:  unaided  browsing,  leveraging  corporate  knowl¬ 
edge  and  experience,  and  computer-aided  techniques  including  reverse  engineering.  In  ad¬ 
dition,  program  understanding  according  to  [Til98]  includes  the  identification,  manipula¬ 
tion,  and  exploration  of  artifacts  of  a  “particular  representation  of  a  subject  system  via  men¬ 
tal  pattern  recognition  by  the  software  engineer  and  the  aggregation  of  these  artifacts  to 
form  more  abstract  system  representations”  [Til98], 

Unaided  browsing  techniques  normally  employed  in  the  process  include  re¬ 
viewing  the  source  code  [HCM02];  however,  this  method  becomes  unwieldy  as  the  lines  of 
code  exceed  nonnal  limits  of  manual  comprehension.  Leveraging  corporate  knowledge 
and  experience  is  also  a  valuable  method,  although  maybe  of  limited  value  under  certain 
conditions  including: 

•  The  lack  of  available  primary  source  corporate  knowledge, 

•  Third-party  system  acquisitions, 

•  Outsourcing. 

Computer-aided  reverse  engineering  methodologies  304  is  a  third  method  of 
addressing  some  of  the  shortcomings  of  the  two  previous  support  mechanisms  categories 
addressed  by  [Til98].  [JBL97,  SLB+98,  and  BLS+99]  contributed  a  body  of  knowledge 
gennane  to  this  research  initiative  with  research  on  the  Janus  model  at  the  Naval  Post¬ 
graduate  School  (NPS).  Follow-on  NPS  reverse-engineering  initiatives  including  research 
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Reverse  engineering  is  the  process  of  understanding,  analyzing,  and  abstracting  the  system  to  a  form  at  a  higher  level  of  abstraction 
[01s95], 

303 

Program  understanding  is  a  relatively  new  area  of  study  with  an  evolving  terminology  and  focus,  with  an  objective  to  acquire  suffi¬ 
cient  knowledge  about  a  software  system  so  that  it  can  evolve  in  a  disciplined  manner.  The  essence  of  program  understanding  is  to  iden¬ 
tify  artifacts  and  understand  the  relationships,  and  is  analogous  to  pattern  matching  concepts  at  various  abstraction  levels  [Til98], 
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A  reverse  engineering  environment  can  manage  the  complexities  of  program  understanding  by  helping  the  software  engineer  extract 
high-level  information  from  low-level  artifacts,  such  as  source  code,  reduces  the  level  of  tedious,  error-prone,  manual  efforts  [Til98], 
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by  [SLB+99,  WS99,  LBS01]  employed  all  three  support  mechanism  categories  identified 
by  [Til98],  and  concluded  that  re-engineering  using  the  combination  of  reverse  engineering 
(e.g.,  extracting  the  most  useful  infonnation)  with  forward  engineering  techniques  ad¬ 
dressed  by  [SLB01]  using  computer-aided  prototyping  techniques  (CAPS)  were  cost- 
effective  methods  for  re-engineering  legacy  M&S  software.  Recent  NPS  research  by 
[You02c]  and  [Pru03]  support  both  the  architectural  mining  and  reengineering  activities, 
and  the  follow-on  simulation  software  application  efforts. 

In  addition,  software  evolution  initiatives  using  prototype  languages  such  as 
the  Prototype  System  Description  Language  (PSDL)  introduced  by  [Luq89,  Luq90]  and  the 
Computer  Aided  Prototyping  System  (CAPS)  development  environment  [LK88,  BL94, 
Luq92,  Luq94,  Luq95,  Luq98]  support  the  rapid,  accurate,  and  timely  development  of  pro¬ 
totypes.  Automated  tools  [WK98,  LBS01,  Fav02]  provide  a  capability  to  invent,  correct 
and  refine  the  conceptual  models  for  new  system  architectures.  This  research  expands 
upon  the  previous  NPS  work  in  reverse-engineering  and  forward-engineering  using  an  en¬ 
tire  Department  M&S  domain  model  as  its  focus. 

Different  reverse-engineering  tasks  [Til98,  YD01,  YK02]  include  program 
analysis,  syntactic  pattern  matching  in  the  programming  language  domain;  plan  recogni¬ 
tion,  semantic  pattern  matching  [SR02]  in  the  programming  language  domain;  concept  as¬ 
signment  semantic  pattern  matching  in  the  application  domain;  redocumentation306,  aggre- 

OAT 

gation,  rejuvenation  and  reconfiguration  of  assets,  and  architecture  recovery  .  [Til98, 
BBW+99]  also  cautions  that  reverse  engineering  and  the  associated  terminology,  tools 
[DFP+02],  and  techniques  are  still  inexact  and  relatively  immature. 

Software  engineers  addressed  program  understanding  in  several  cognitive 
models  [SFM97].  [Til98]  explains  two  common  approaches  to  program  understanding, 
based  on  the  software  engineer’s  level  of  domain  expertise:  the  functionally  based  bottom- 
up  deductive  approach  focused  on  the  cognition  of  the  implementation  domain  and  what 
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Forward  engineering  is  the  set  of  engineering  activities  that  consume  the  products  and  artifacts  derived  from  the  legacy  software  and 
new  requirements  to  produce  a  new  target  system  [01s95], 

306  Redocumentation  is  the  process  of  retroactively  providing  documentation  for  an  existing  system  [Til98], 
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Architecture  recovery  or  structural  redocumentation  is  a  term  for  using  reverse  engineering  to  reconstruct  the  architectural  aspects  of 
software  [Til98], 

308  A  cognitive  model  describes  the  cognitive  processes  and  knowledge  structures  used  to  form  a  mental  presentation  of  the  program 
being  studied  [SFM97], 
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the  system  does;  and  the  top  down,  inductive  behavioral  approach,  emphasizing  how  the 
system  works,  based  on  an  existing  notion  of  the  system  functionality  and  application  do¬ 
main  and  employing  a  goal-driven  method  of  hypothesis  postulation  and  refinement  on  ex¬ 
pected  artifacts.  Case  studies  according  to  [Til98],  show  that  in  practice,  software  engi¬ 
neers  switch  between  these  two  models  employing  a  third  model,  the  opportunistic  ap¬ 
proach,  depending  on  the  problem  at  hand.  Knowledge  management  techniques  [SG96b, 
KNR+01,  KMS02,  WJC02,  RJ02,  Lie02,  RL02,  Wel02]  provide  possible  approaches  for 
reusing  software  engineering-related  knowledge  and  program  understanding. 

[Til98]  also  forwards  the  idea  that  reverse  engineering  as  the  predominant 
support  mechanism  used  to  support  program  understanding  is  an  activity,  which  does  not 
change  the  subject  system,  since  it  is  an  examination  process  vice  an  alteration  process  op¬ 
timally  employed  to  identify  artifacts,  discover  relationships,  and  generate  abstractions. 

The  process  is  dependent  on  several  variables  including  cognitive  ability  and  preferences, 
domain  familiarity  and  supporting  facilities  to  understand  the  three  categories  of  artifacts 
identified  by  [Til98] :  data,  knowledge,  and  information309. 

Three  canonical  reverse-engineering  activities  support  the  manipulation  of 
these  artifacts:  data  gathering  [Til98],  knowledge  management,  and  information  explora¬ 
tion,  including  navigation,  analysis  cited  by  [You89].  [BM99  and  Moo02]  view  reverse¬ 
engineering  methods  as  a  necessary  step  before  starting  software  engineering  activities  for 
an  improved  system.  [TK02]  believe  a  correctly  implemented  reverse  engineering  process 
may  directly  benefit  follow-on  reengineering  activities. 

2.  Product  Development  of  the  Software  Product  Line  Architecture 

a.  Reengineering  Architecture  Assets  for  Software  Product  Lines 

Software  systems  have  become  larger,  more  complex,  more  costly  and 
longer  lived,  which  challenges  the  early  software  life-cycle  models  that  modeled  systems 
maintained  for  a  short  time,  until  they  were  retired  or  replaced.  The  software  engineering 
challenge  cited  by  [WNS+97]  is  how  to  move  a  large  body  of  legacy  code  from  its  current 
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Data  is  the  factual  information  used  as  the  basis  for  study,  reasoning  or  discussion.  Knowledge  is  the  sum  of  what  is  known,  which 
includes  data  and  information  such  as  relationships  and  rules  progressively  derived  from  the  data.  Information  is  contextually  and  selec¬ 
tively  communicated  knowledge.  [Til98] 
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state  to  a  condition  in  which  it  can  evolve  in  a  disciplined  way.  Existing  software  systems 
have  several  inherent  problems,  which  adversely  affects  maintenance  [OS93] : 

310 

•  Legacy  systems  are  usually  complex,  unstructured,  highly  coupled  ,  with  low  co¬ 
hesion311, 

•  Maintenance  can  create  unpredictable  ripple  effects, 

•  Documentation  is  often  missing,  incomplete,  outdated,  or  unreliable, 

•  The  system  is  obsolete  with  unsupported  hardware  or  software  components, 

•  Experienced  software  engineers,  programmers,  and  software  maintainers  are  hard  to 
keep, 

•  Maintenance  backlogs  continue  to  grow  [OS93]. 

[OS93]  defines  reengineering  as  the  bridge  to  an  organization’s  newly  de¬ 
fined  processes  and  environment  from  the  legacy  software  system.  Reengineering  is  often 
a  better  option  than  redeveloping  the  system  when  the  following  factors  are  considered: 

•  Knowledge  is  imbedded  in  the  software  logic, 

•  Reengineering  allows  an  organization  to  recoup  its  investment  of  time,  money  and 
knowledge, 

•  Legacy  software  is  a  valuable  organization  asset  and  reengineering  extends  the  life 
of  these  systems. 

•  Reengineered  /  reused  code  costs  less  than  redeveloped  code  [OS93]. 

Figure  10-11  illustrates  the  selected  software  architecture  reconstruction  ap¬ 
proach.  Reengineering  has  the  potential  to  improve  an  organization’s  understanding  of  the 
software,  establish  conditions  to  improve  future  versions  [01s93,  01s95,  BNS97,  WBS+97, 
WNS+97,  BSW99,  BSW+99,  HDK+00,  YD01]  and  better  understand  past  failures 
[BST+99].  In  many  existing  software  systems,  the  projects  were  conceived,  designed,  and 
developed  as  unique  products,  with  minimal  integration,  and  very  little  systematic  reuse  of 
assets,  which  [WBS+97]  suggest  lose  value  over  time  by  getting  stale  and  requiring  more 
assets  to  maintain  them. 

Replacing  these  systems  and  the  significant  investment  they  represent  with 
new  systems,  requiring  a  new  major  investment  is  unlikely,  notes  [WBS+97]  while  con¬ 
tinuing  a  fine-grained  maintenance  strategy,  expecting  the  systems  to  evolve  into  maintain¬ 
able  assets,  appears  overly  optimistic.  In  addition  to  the  traditional  software  maintenance 
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Coupling  is  the  measure  of  interconnection  among  modules  in  a  program  structure.  The  lowest  possible  coupling  is  desired  [Pre97]. 

31 1 

A  cohesive  module  ideally  performs  a  single  task  within  a  software  procedure,  requiring  little  interactions  with  other  parts  of  the 
program.  High  cohesion  is  desired  [Pre97] 
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activities,  an  assessment  of  the  legacy  system  may  suggest  a  replacement  strategy,  a  trans- 
formation  strategy,  either  white-box  “  or  black  box  ,  or  a  combination  strategy 
[WNS+97]. 

[WNS+97]  suggest  that  time-to-market  and  widespread  distribution  tech¬ 
nology  employing  the  Web  and  browsers  will  drive  large-grained  use  of  legacy  system 
components  integrated  by  middleware  and  object-oriented  technologies,  as  opposed  to 
starting  development  from  scratch.  Furthermore,  [WBS+97]  note  that  economic  realities 
are  making  low-level  maintenance  activities  unattractive  when  compared  with  the  potential 
gains  from  large-scale  reuse,  supported  by  a  focus  on  architecture  and  product  lines  em¬ 
ploying  middleware  and  wrapping  technologies.  However,  [BST+99],  caution  that  several 
organizational,  management,  and  technical  pitfalls  may  befall  a  reengineering  project: 

•  A  flawed  or  incomplete  reengineering  strategy, 

•  Inappropriate  use  of  outside  consultants  and  outside  contractors, 

•  Work  force  inadequately  trained  or  tied  to  old  technologies, 

•  Legacy  system  not  under  control, 

•  Requirements  elicitation  and  validation  inadequate, 

•  Software  architecture  is  not  a  primary  reengineering  consideration, 

•  No  notion  of  a  separate,  discrete  reengineering  process, 

•  Inadequate  planning  or  discipline  to  execute  the  plan, 

•  Management  lacks  long-term  commitment, 

•  Management  predetennines  technical  decisions  [BST+99]. 

In  recognition  that  modern  organizations  own  or  use  a  significant  software 
portfolio,  representing  a  major  investment  of  organization  resources,  the  challenge  high¬ 
lighted  by  [BSW+99]  is  to  appreciate  the  value  of  the  organization’s  software  portfolio, 
and  reduce  the  liability  of  software  asset  depreciation  over  time.  The  emerging  emphasis 
on  software  architectures  and  evolution  of  software  systems  provides  an  impetus  to  the  de¬ 
velopment  of  product  lines,  since  systems  with  interoperable  architectures  allow  the  soft¬ 
ware  to  operate  across  defined  interfaces,  although  [BSW+99]  notes  that  legacy  systems 
must  be  updated  to  allow  them  to  interact  as  well-behaved  components. 
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White-box  transformation  includes  program  understanding  consisting  of  activities  to  recover  lost  structure  or  documentation 
[WNS+97], 

313 

Black-box  transformation  includes  wrapping  or  encapsulating  the  legacy  system  based  on  an  understanding  of  the  external  interfaces, 
without  trying  to  understand  the  internal  structure  [WNS+97]. 
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Figure  10-11.  Software  Architecture  Reconstruction 

b.  Reengineering  for  Software  Product  Line  Architecture  Variability 

Recall  from  Chapter  IX  that  the  five  BMDS  missile  domain  System-Level 
M&S  possessed  two  hundred  and  eighty- five  major  simulation  model  representations,  with 
only  a  sixty  percent  availability  of  authoritative  model  representations.  Note,  this  statistic 
only  accounts  for  the  specific  major  system  model  representations  detailed  in  Tables  9-1 
through  Table  9-14,  and  does  not  include  other  model  representations  resident  in  the  five 
simulations,  nor  does  it  factor  in  the  different  geodetic  environments. 

It  is  significant  to  note  that  the  lack  of  authoritative  model  representations  is 
not  apparent  from  a  single  simulation  viewpoint.  However,  when  viewed  as  part  of  a  do¬ 
main  hierarchy,  against  the  same  standard  for  an  authoritative  model  representation,  both 
the  shortfalls  and  the  patterns  identified  in  Table  9-1  through  Table  9-14  in  Chapter  IX  be¬ 
come  readily  apparent.  These  shortfalls  and  patterns  may  contribute  to  interoperability 
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anomalies  discussed  earlier.  This  condition  is  the  current  “as-is”  state  of  the  five  BMDS 
missile  domain  System-Level  M&S. 

The  scope  of  the  future  BMDS  mission  is  unprecedented.  The  illustration  in 
Figure  10-12  notionally  represents  the  future  “to-be”  requirements  for  the  BMDS  missile 
domain  System-Level  M&S,  with  eleven  possible  dimensions  the  architectural  approach 
will  support.  The  future  requirements  for  the  BMDS  missile  domain  System-Level  M&S 
may  add  dimensions  and  significant  complexity  including  possible  variant,  version,  feature, 
quality,  and  option  dimensions  of  future  systems,  in  addition  to  the  existing  four  dimen¬ 
sions  of  abstraction  (e.g.,  structural,  functional,  temporal,  and  qualitative).  The  future 
BMDS  requirements  also  add  two  more  dimensions  for  the  BMDS  System-Level  M&S,  the 
cyclic  life  cycle  support  dimension  and  the  quantitative  dimension.  The  cyclic  life  cycle 
dimension  support  for  all  BMDS  Element-  and  System-Level  organizations  occurs  concur¬ 
rently  during  all  three  life  cycle  phases  (e.g.,  research  and  development,  transition,  and  op¬ 
erational  support).  The  quantitative  dimension  suggests  the  need  for  quantitative  meas¬ 
urements  to  augment  qualitative  identifiers,  when  possible,  for  model  representation  devel¬ 
opment. 

Existing  BMDS  element  systems  model  representations  must  closely  mirror 
the  existing  system  and  keep  current  with  new  capabilities,  normally  introduced  through 
component  software  upgrades,  suggesting  a  close  relation  between  the  many  element  sys¬ 
tem  /  component  software  requirement  specifications  and  the  component  model  software 
build  schedules.  The  current  BMDS  model  representations  may  be  adequate  for  some  fu¬ 
ture  requirements.  However,  as  the  number  of  BMD  system  level  components  continue  to 
grow  and  different  manufacturers,  or  in  the  future,  different  countries,  provide  different  ca¬ 
pabilities  (e.g.,  radars,  boosters,  or  kill  vehicles),  the  BMDS  requires  the  capability  to  as¬ 
sess  the  contribution  of  that  component  to  the  overall  BMD  system  level  capability. 

Although  temporal  and  life  cycle  requirements  for  these  future  systems  are 
still  unknown,  the  BMDS  may  require  one  or  more  model  representations  for  the  six  blocks 
from  Block  04  through  Block  14  to  support  temporal  dimension  requirements.  Based  on 
the  current  population  of  two  hundred  and  eighty- five  model  representations,  for  planning 
purposes  one  model  representation  developed  for  each  future  block  period  would  require 
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the  near-term  development  of  approximately  one  thousand  model  representations.  The 
blue  right-to-left  vertical  line  graphics  annotated  with  element  names  and  cited  in  the  leg¬ 
end,  represent  this  population  in  Figure  10-12. 


Life  Cycle  M&S  Support 


Notional  Architecture  Of  Missile  Defense  Systems  Model  Representation  Requirements 


Figure  10-12.  Future  BMDS  Requirements  (From  [Gre03b]) 

The  BMDS  life  cycle  support  dimension  is  currently  undetermined  and  un¬ 
der  study.  Recall  that  the  Agency  built  the  current  five  simulations  under  study  to  primar¬ 
ily  support  only  the  missile  defense  research  and  development  phase.  The  emerging 
BMDS  will  also  support  the  transition  and  operational  support  phases  including  new  func¬ 
tional  requirements  for  training,  logistics,  mission  planning,  system  maintenance,  and  op¬ 
erational  planning.  Assuming  for  planning  purposes  these  new  functions  generate  an  addi¬ 
tional  single  model  representation  requirement  per  simulation  per  block,  an  additional  one 
thousand  representations  may  be  required.  The  overall  requirement  for  future  BMDS 
model  representations,  based  on  these  assumptions  indicates  a  potential  need  for  future 
BMDS  model  representations  numbering  in  the  thousands. 
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The  BMDS  system  engineering  functions  requires  flexible,  responsive  System- 
Level  M&S  with  the  ability  to  provide  here-to-for  unknown  future  variants,  versions,  fea¬ 
tures,  and  option  requirements  for  spiral  development  within  the  blocks,  analysis  of  alterna¬ 
tives  for  new  systems,  and  trade  space  analysis  for  future  systems  or  new  capabilities.  The 
illustration  in  Figure  10-12  identifies  these  notional  top-level  requirements. 

The  past  performance  of  the  Department’s  and  the  Agency’s  development 
processes,  practices,  and  techniques  cited  in  Chapters  II  through  Chapter  V  proved  less 
than  satisfactory  for  previous  model  representation  requirements,  and  it  is  questionable 
whether  exhortations  for  “better,  faster,  cheaper”  development  and  V&V  practices,  based 
on  the  current  practices  will  be  sufficient  to  meet  projected  requirements  cited  above.  The 
research  strongly  suggest  the  previous  method  of  individual  system-based  development  and 
low-levels  of  reuse,  may  not  support  optimal  development  methodologies  for  the  future 
BMDS.  In  this  effort  we  chose  the  domain  management  level  as  the  optimal  level  to  exe¬ 
cute  such  a  program,  since  multiple  Department  enterprise  efforts  (e.g.,  JWARS,  JSIMS, 
JMASS)  were  less  than  successful,  and  management  efforts  below  the  domain  level  may 
lack  the  resources  and  perspectives  essential  to  the  task 

The  selected  approach  involves  the  introduction  of  an  additional  level  of  ab¬ 
straction  beyond  the  current  software-  or  component-level  into  a  software  architecture 
framework  allowing  the  visibility  of  assets  and  the  ability  to  leverage  feature314  commonal¬ 
ity.  The  definition  of  a  feature  presented  by  [BosOO]  emphasizes  the  early  inclusion  of 
quality  attributes  discussed  in  Chapter  VI.  Features,  variants,  versions,  options,  and  quality 
attributes,  with  associated  profiles  ,  will  be  stored  in  the  Simulation  Software  Architec¬ 
ture-Based  Product  Line  Model  Domain  Metadata  Repository  illustrated  in  Figure  10-13. 

The  Simulation  Software  Architecture-Based  Product  Line  Model  Domain 
Metadata  Repository  maintains  the  feature,  variant,  version,  option,  and  quality  attributes  at 
the  software  product  line  architecture  level,  and  supports  management  of  the  product  fami¬ 
lies  (e.g.  weapons,  sensors),  software  product  lines  (e.g.  missiles,  radars),  and  software 
product  line  systems  (e.g.  PAC-3,  THAAD  missiles)  at  the  domain  level  with  a  top-down 
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[BosOO]  defines  a  feature  as  a  logical  unit  of  behavior  that  is  specified  by  a  set  of  functional  and  quality  requirements. 
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A  profile  is  a  set  of  scenarios  that  may  be  used  as  the  basis  for  specifying  a  number  of  quality  attributes  such  as  performance  or  reli¬ 
ability  [BosOO], 
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management  approach,  and  decentralized  execution  supporting  the  concept  of  software  ar¬ 
chitecture-based  development. 


E.  THE  SIMULATION  SOFTWARE  ARCHITECTURE-BASED  PRODUCT 
LINE  MODEL  DOMAIN  METADATA  REPOSITORY 

1.  The  Simulation  Software  Architecture-Based  Product  Line  Model  Do¬ 
main  Metadata  Repository 

The  Simulation  Software  Architecture-Based  Product  Line  Model  Domain  Meta¬ 
data  Repository  in  Figure  10-13  provides  a  public  interface  and  a  defined  set  of  services 
with  layer  usage  dependent  on  lower  layers.  The  SSA  model  permits  a  few  exceptions  for 
unique  applications.  The  Simulation  Software  Architecture -Based  Product  Line  Model 
Domain  Metadata  Repository  (e.g.,  Metadata  Repository)  supports  all  four  layers  of  the 
SSA  Model. 


ARCHITECTURE 


-BASED 

PRODUCT 


Layer  0  -  The  Real  World 


Figure  10-13.  SSA-Based  Product  Line  Model  Domain  Metadata  Repository 
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The  primary  purpose  for  the  Metadata  Repository  is  to  address  the  root  cause  of 
heterogeneous  system  representation  and  data  anomalies  explained  in  Chapter  IV.  Major 
components  of  the  Simulation  Software  Architecture-Based  Product  Line  Model  Domain 
Metadata  Repository  tailored  from  [IS079c]  provides  the  interfaces  shown  in  Figure  10-14 
and  includes: 

•  A  Domain  Metadata  Registry  holding  infonnation  describing  the  structure,  format 
and  definition  of  data  [Ste03b].  XML  recently  emerged  as  the  industry 
data/metadata  interchange  format  of  choice  [DavOl]  and  mandated  by  the  Depart¬ 
ment’s  Joint  Technical  Architecture  [JTA02a,  JTA02b]  for  domain  and  application 
uses  of  specific  markup  languages  defined  by  tagged  data  items  [ASC02].  [ASC02] 
also  identifies  the  requirements  for  a  Department-level  XML  Registry  (DXR) 
[Ste03a]  and  Clearinghouse  supporting  the  development  of  a  Federal-level  XML 
Registry  (FXR)  [Ste03b,  XWG02]  with  components  developed  with  the  appropriate 
XML  Namespace  [Sal02].  The  Domain  Metadata  Registry  will  support  the  follow¬ 
ing  emerging  XML  standards:  XML  query  (XQuery)  data  models,  algebra,  and 
query  language  [CFM+03];  the  XQuery  1.0  and  XPath  2.0  Data  Model  [FMM+03]; 
and  XQuery  1.0  and  XPath  2.0  Formal  Semantics  [FMM+03], 

•  A  Domain  Metadata  Catalog  containing  instances  of  metadata  associated  with  do¬ 
main  data  resources,  supporting  the  use  of  search  portals  and  queries  exploring  the 
Domain  Metadata  Catalog  to  locate  relevant  data  (e.g.,  data  dictionary), 

•  A  Domain  MetaClass  Catalog,  which  supports  the  development  of  metamodels 
with  a  class  whose  instances  are  classes, 

•  A  Metalanguage  Catalog  which  specifies  some  or  all  of  the  aspects  of  a  MetaLan- 
guage  used  in  the  Domain  (e.g.,  Backus-Naur  form,  UML,  ADL,  XML), 

•  A  Domain  Meta-Metamodel  Catalog  or  model  that  defines  the  language  for  ex¬ 
pressing  a  metamodel, 

•  A  Domain  MetaModel 316  Catalog  defining  the  language  for  expressing  a  mode, 

•  Domain  MetaObject  Catalog  includes  all  metaentities  in  a  metamodeling  language 
including  metatypes,  metaclasses,  metaattributes,  and  metaassociations  [DB03], 

•  Domain  Ontologies  Catalog  including  data  categorization  schemes,  thesauruses, 
glossaries,  key-word  lists,  and  taxonomies,  supporting  semantic  and  syntactic  un¬ 
derstanding  heterogeneous  system  representation  data, 

•  Domain  Schemas'  Catalog  representing  database  tables  and  relationship  struc¬ 
tures,  XML  document  type  definitions  (DTD),  and  XML  schema, 

•  Features,  versions,  variants,  options,  and  quality  attributes,  with  associated  pro¬ 
files, 

•  Architecture  Readiness  Levels  (ARLs)  discussed  later  in  the  chapter. 


316  A  metamodel  is  a  model  that  describes  other  models  [IS079c]. 
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Schemas  include  diagrams,  outlines,  or  models  representing  data  structure,  organization,  format,  structure,  relationship,  or  data. 
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Figure  10-14.  BMDS  Domain  Metadata  Repository  (DMR) 

2.  The  Simulation  Software  Architecture-Based  Product  Line  Model  Do¬ 
main  Metadata  Repository  Registry 

The  Simulation  Software  Architecture-Based  Product  Line  Model  Domain  Meta- 

o  1  o 

data  Repository  Registry  (e.g.,  Domain  MRR  or  DMRR)  structure  in  Figure  10-15  uses  a 
conceptual  data  model  (e.g.,  registry  metamodel)  to  maintain  information  about  data  ele¬ 
ments  and  associated  concepts  (e.g.,  conceptual  domains,  value  domains),  including  the 
metadata  items  described  above.  The  DMRR  maintains  instances  of  domain  metadata 
items,  which  define  types  of  application  level  data  (e.g.,  in  a  relational  database  schema), 
subsequently  populated  with  real  world  data.  A  Unified  Modeling  Language  (UML)  subset 


318  A  metadata  registry  maintains  an  information  system  for  registering  metadata  [IS079c]. 

319 

A  conceptual  data  model  represents  an  abstract  view  of  the  world.  A  data  model  is  a  graphical  and/or  lexical  representation  of  data, 

specifying  their  properties,  structure  and  inter-relationships  [lS079c]. 

320 

Metadata  items  are  an  instance  of  a  metadata  object  [IS079c] 
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from  [IS079c]  in  Appendix  J.  specifies  the  registry  metamodel’21  including  metamodel 
constructs  ~  for  classes,  relationships,  association  classes,  attributes  "  ,  composite  attrib- 
utes  ,  and  composite  data  types  .  [IS079c]  uses  the  term  “metamodel  construct”  for  the 
model  construct  it  uses,  and  the  term  “metadata  objects”  for  the  model  constructs  it  speci¬ 
fies. 


Layer  0  -  The  Real  World 


Figure  10-15.  BMDS  Domain  Metadata  Repository  Registry  (DMRR) 


The  DMRR  metamodel  specifies  types  of  classes,  attributes,  and  relationships.  A 
specific  of  classes,  attributes,  or  relationship  instance  will  be  a  specific  type,  and  at  any 
point  in  time  will  contain  a  specific  value.  The  DMRR  metamodel  will  be  extensible,  with 
the  ability  to  add  future  classes,  relationships,  and  attributes  to  the  conceptual  data  model. 


A  metamodel  specifying  a  metadata  registry  [IS079c]. 

322 

A  unit  of  measuring  for  modeling  [IS079c], 

323 

A  characteristic  of  an  object  or  class  [IS079c]. 

324 

An  attribute  whose  datatype  is  non-atomic  [IS079c] . 

325 

A  datatype  that  is  also  a  class  [IS079c], 
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The  DMRR  provides  a  unified  view  of  concepts,  terms,  value  domains,  and  value 
meanings326;  promotes  a  unifying  view  of  the  data  holdings;  and  enables  reuse  and  data 
sharing  by  providing: 

•  A  means  for  coordinating  data  requirements  between  customers  and  systems  that 
store,  exchange,  or  manipulate  data, 

•  Assistance  to  registrars  "  maintaining  consistency  among  different  registries, 

•  A  means  to  store,  manipulate,  and  exchange  metadata  in  support  of  data  attribution, 
classification,  definition,  naming,  identification,  and  registration, 

•  A  consistent  content  supporting  interoperability, 

•  Schema  mappings  of  each  tool  set, 

•  Support  to  translating  constructs  into  the  different  languages, 

•  Preservation  of  the  concepts  maintained  in  the  original  model, 

•  A  conceptual  model  to  base  the  development  of  specific  logical  model  (e.g.,  model 
of  the  information  system)  in  an  infonnation  system  (e.g.,  database  design)  for  the 
required  application  [IS079c]. 

F.  ARCHITECTURE  READINESS  LEVELS  (ARL) 

A  persistent  systemic  problem  facing  decision-makers  is  the  amount  of  confidence 
to  place  on  simulation  results.  Research  suggests  this  issue  has  many  dimensions,  as  noted 
in  the  previous  chapters.  One  of  the  issues  involves  the  credibility  of  the  model  representa¬ 
tion  or  simulation  component.  Software  reuse  and  component-based  strategies  consistently 
faced  challenges  establishing  the  credibility  of  the  software  component  (e.g.,  predictable 
composition). 

Composable  solutions  necessitate  the  need  for  defined  quality  properties  and  a  basis 
for  predicting  the  quality  properties.  A  major  challenge  of  the  current  software  engineering 
process  is  applying  software  quality  models  and  developing  reliable  software  productivity 
metrics  [IEE92]  during  the  early  design  and  analysis  phases  of  the  project  development. 
Chapter  VIII  provided  a  discussion  of  software  architecture  quality  analysis  methods.  In¬ 
terfaces  pose  special  concerns  for  future  component-based  strategies  [BBB+00]. 

Similar  problem  in  the  technology,  manufacturing,  and  integration  sectors  resulted 
in  process  models  devised  as  systematic  metric  and  measurement  systems,  employed  to 
compare  the  maturity  between  different  types  of  technology  [ACH+02].  Three  Readiness 


3-6  The  meaning  or  semantic  content  of  a  value  [IS079c]. 

327 

A  representative  of  the  Registration  Authority,  which  is  the  organization  responsible  for  maintaining  a  register  [IS079c}. 
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Level  models  illustrated  in  Figure  10-16  support  the  assessment  of  technology  maturity, 
production  readiness  maturity,  and  integration  readiness  maturity  for  a  specific  technology, 
product  or  process  technologies:  Technology  Readiness  Levels  (TRLs)  [Man95],  Engineer¬ 
ing  Manufacturing  Readiness  Levels  (EMRLs)  [FioOl],  and  Integration  Readiness  Level 
(IRLs)  [MDA02z].  NASA  space  technology  planning  employed  Technology  Readiness 
Levels  (TRLs)  for  many  years  [Man95]. 


TRL 

TECHNOLOGY  READINESS 

LEVELS  (TRLs)  [Man951 

EMRL 

ENGR/MFG  READINESS 

LEVELS  (EMRLs)  [FioOl] 

SYSTEM  /  COMPONENT  /  ITEM  (sci) 

IRL 

INTEGRATION  READINESS 

LEVELS  (IRLs)  [MDA02Z| 

9 

ACTUAL  SYSTEM  ‘FLIGHT 
PROVEN’  IN  SUCESSFUL 
MISSION  OPERATIONS 

TECHNOLOGIES 

MUST  BE 
MATURED  TO 
AT  LEAST  TRL  4 
OR  5  BEFORE  IT 
IS  READY  FOR 
PRODUCTION 

9 

TECHNOLOGY  PROVEN  IN 
SYSTEM  OPERATIONAL  TEST 
/  INTEGRATED / PROVEN 

OC 

ACTUAL  SYSTEM  COMPLETE 
AND  FLIGHT  QUALIFIED  IN 
TEST  AND  DEMONSTRATION 

8 

ACTUAL  SYSTEM  COMPLETE 
&  QUALIFIED  BY  TEST  AND 
DEMONSTRATION 

7 

SYSTEM  PROTOTYPE 
DEMONSTRATED  IN  OPER- 

ATIONL  ENVIRONMENT 

7 

INTEGRATION-READY  TECH 
DEMO-COMP/ELEMENT 
INTEGRATED  AND  DEMO 

6 

SYSTEM/SUBSYSTEM  MODEL 
OR  PROTOTYPE  DEMO  IN 

RELEVANT  ENVIRONMENT 

6 

TECHNOLOGY  ADAPTION 
COMPONENT  F3  DEMO  IN 
SYSTEM  ARCHITECTURE 

5 

COMPONENT / BREADBOARD 

VALIDATED  IN  RELEVANT 

ENVIRONMENT 

5 

IDENTICAL  SYSTEM  / 

COMP  /  ITEM 

PRODUCEDOR  S/C/I  PROD. 

H 

SYSTEMS  ENGINEERING  AND 
ANALYSIS-FUNCTIONS  VAL 

BY  PROTOTYPE 

4 

COMPONENT / BREADBOARD 

VALIDATED  IN  LAB 

4 

SYSTEM  /  COMP  /  ITEM  IN 
PRODUCTION  OR  LRIP. 
READY  FOR  FULL  PROD 

COMPONENT  F3 PROVEN 

FEASIBLE  IN  LAB 

3 

ANAL / EXPERIMENTAL 
CRITICAL  FUNC/CHAR  PROOF 
OF  CONCEPT 

3 

SYSTEM  /  COMP  /  ITEM  IN 

ADV  DEV.  READY  FOR 

LOW  RATE  PRODUCTION 

SYSTEM  APLLICATION  AND 
INITIAL  INTERFACE  DOCS 
DEVELOPED 

2 

TECHNOLOGY  CONCEPT  OR 

APPLICATION  FORMULATED 

2 

SYSTEM  IN  PRTOTYPE  DEMC 

BEYOND  BRASS  BOARD  DEV 

2 

INTERFACE  FORM/FIT/ 
FUNCTION  (F3)  DEVELOPED 

1 

BASIC  PRINCIPLES  OBSERVED 

AND  REPORTED 

1 

SYSTEMVALIDATED  IN  LAB 

OR  EARLY  DEVELOPMENT 

1 

BASIC  APPLICATION  TO 
SYSTEM  OBSERVED  AND 
REPORTED _ 

Figure  10-16.  Current  Readiness  Level  Methods 

A  major  element  of  the  M&S  product  line  architecture  approach  introduced  in  this 
research  is  the  Architecture  Readiness  Level  (ARL)  process,  the  simulation  software  archi¬ 
tecture  equivalent  to  the  TRL,  EMRL,  and  IRL  processes.  The  ARL  has  three  major  objec¬ 
tives:  1)  measure  software  quality  early  in  the  development  process  as  part  of  the  verifica¬ 
tion  process,  2)  identify  and  measure  M&S  quality  attributes  as  part  of  the  V&V  process, 
and  3)  identify  and  measure  the  quality  of  data  quality  attributes  supporting  the  validation 
process.  The  ARL  will  be  employed  in  all  life  cycle  phases  of  the  simulation  product  line 
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development  and  employment  including:  1)  mining,  2)  reverse  engineering,  3)  reengineer¬ 
ing,  4)  development,  and  5)  VV&A.  The  ARL  supports: 

•  Model  representation  quality  attributes, 

•  Simulation  quality  attributes, 

•  Data  quality  attributes 

•  The  Department’s  VV&A  process. 

The  proposed  ARL  for  improving  authoritative  system  representations  incorporates  a  stan¬ 
dard  reference  for  evaluating  model  representations  at  any  level  of  abstraction. 


ARL 

LC 

Architecture  Readiness  Level  Summary 

9 

OP 

Verification  and  Validation  completed  successfully. 

Product  Operationally  Integrated  in  Final  Version. 

8 

SE- 

OP 

System  Implementation  Complete.  Validated  data 

Available.  Implementation  Validated 

7 

SE 

Components  and  Interfaces  Composed.  Integration  Testing 

Completed.  Components  and  Interfaces  meet  verified  specs. 

6 

SE 

System  /  Interface  Analysis  and  Design.  Design  Specs  Verified 

Against  Conceptual  Model.  Validation  Data  Confirmed. 

5 

SE- 

SA 

Conceptual  Model  (CM)  Developed.  Verification  and  Validation 

Plan  Developed.  Data  requirements  determined.  CM  Validated 

4 

SA 

System  Views  Developed.  Implementation  Layer  established. 

Product  Determined.  Metadata  employed  from  Registry. 

3 

SA 

Component  Views  Developed.  Component  Layer  established. 

Product  Line  Determined.  Metadata  registered. 

2 

SA 

Conceptual  Views  Developed.  Conceptual  Layer  established. 

Product  Family  Determined.  Metadata  analyzed. 

1 

SA- 

RW 

Viewpoints  /  Views  Developed  from  Real  World  /  Core  Assets. 

Referent  Layer  established.  Metadata  needs  identified. 

Table  10-1.  Architecture  Readiness  Level  Summary 
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Recall  our  earlier  discussion  about  the  different  viewpoints  and  views  of  a  house 
envisioned  by  the  plumber,  electrician,  and  other  craftsmen  contributing  to  the  construc¬ 
tion.  Although  they  may  have  a  primary  viewpoints  and  views  of  the  housed  based  on  their 
craft,  they  share  several  constraints  such  as  the  architectural  drawings;  local-,  state-,  and 
national-level  building  codes;  the  cost  and  availability  of  certain  materials;  and  the  avail¬ 
ability,  experience  level,  and  wage  requirements  of  helpers. 

To  ensure  compliance  with  applicable  building  codes,  the  building  contractor  takes 
out  a  permit  and  schedules  several  visits  by  the  appropriate  building  inspectors  during  the 
construction.  During  the  interior  “roughing  in”  inspection  the  building  inspector  will  re¬ 
view  and  test  the  work  of  the  electrician,  plumber,  and  HVAC  specialist  before  the  drywall 
is  installed  to  ensure  building  code  compliance  by  the  different  contractors.  In  many  cases 
the  building  codes  evolved  after  repetitive  tragic  events  caused  injuries,  deaths,  or  the  loss 
of  property.  The  proposed  ARL  for  improving  authoritative  system  representations  im¬ 
poses  similar  constraints  and  incorporates  a  standard  reference  (e.g.,  building  code)  for 
evaluating  model  representations. 

In  Table  10-1  we  present  the  Architecture  Readiness  Level  Summary  with  nine 
ARL  levels  numbered  one  (e.g.,  immature)  through  nine  (e.g.,  mature),  very  similar  to  the 
base  TRL  Model.  In  addition,  the  life  cycle  view  of  the  representation  matures  from  the 
real-world  view  (RW),  through  the  four  layers  of  the  simulation  software  architecture  (SA) 
process,  into  the  simulation  software  engineering  (SE)  process,  and  finally  into  the  opera¬ 
tion  (OP)  and  maintenance  world.  Three  ARL  transition  points  support  the  ARL  model 
maturity  process  at  ARL  layer  1  (RW  to  SA),  ARL  layer  5  (SA  to  SE),  and  at  ARL  layer  8 
(SE  to  OP).  Note  that  in  the  very  similar  EMRL  Model,  technologies  need  to  meet  a  mini¬ 
mum  of  EMRL  Four  or  EMRL  Five  before  it  is  ready  for  production.  The  ARL  Model 
takes  a  very  similar  tack  for  the  development  of  authoritative  representations. 

The  next  section  provides  a  synopsis  of  each  ARL,  including  the  activities  or  capa¬ 
bilities  associated  with  the  specific  ARL: 

ARL  1  -  The  lowest  level  of  the  architecture  readiness  level  and  transition  point 
from  the  real  world  to  the  Simulation  Software  Architecture  (SSA)  referent  layer.  SSA 
Referent  layer  viewpoints  from  the  real  world  lack  definition  and  understanding.  Views 
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and  requirements  are  immature.  Attribute328  analysis  initiated.  Core  asset  analysis  is  in¬ 
complete.  Architectural  components  establishing  the  view  require  significant  descriptive 
and  classification  information.  Metadata  analysis  is  pending.  Architecture  is  immature. 
Quality  requirements  and  measurement  process  initiated.  Validation  data  require¬ 
ments  are  unknown.  The  SSA  is  very  immature  and  very  high  risk. 

ARL  2  -  The  second  level  of  the  architecture  readiness  level,  the  SSA  Conceptual 
Layer  is  established.  Conceptual  viewpoints  from  the  real  world  have  limited  definition 
and  understanding.  Views  and  requirements  more  mature.  Architectural  components  es¬ 
tablishing  the  view  require  additional  descriptive  and  classification  information.  Metadata 
analysis  is  largely  incomplete.  Product  family  analysis  initiated.  Validation  data  analysis 
initiated.  Metrics  framework  and  software  quality  metrics  process  initiated.  Core  as¬ 
set  analysis  is  complete.  Initial  query  sent  to  Domain  or  Enterprise  Metadata  Registry  for 
possible  item  availability  and  reuse.  The  SSA  remains  immature  and  high  risk. 

ARL  3  -  The  third  level  of  the  architecture  readiness  level,  the  SSA  Component 
Layer  is  established.  Component  viewpoints  from  the  Domain  or  Enterprise  Metadata  Reg- 

999 

istry  or  the  SSA  Conceptual  Layer  have  more  definition  and  understanding.  Metaclass 
established.  Views  and  requirements  validated.  Architectural  components  establishing  the 
view  have  adequate  descriptive  and  classification  information.  Metadata  analysis  is  largely 
complete  and  metadata  registered.  Product  line  analysis  initiated.  Product  metrics  estab¬ 
lished.  Validation  data  analysis  is  complete.  Metrics  framework  and  software  quality  met¬ 
rics  process  in  progress.  Core  asset  utilization  options  established.  Results  received  from 
Domain  or  Enterprise  Metadata  Repository  search  for  possible  item  availability  and  reuse. 
The  SSA  is  maturing  and  mitigation  actions  developed  to  address  the  risk(s). 


328  A  measurable  physical  or  abstract  property  of  an  entity  [IEE98d], 

329 

A  requirement  that  a  software  attribute  be  present  in  software  to  satisfy  a  contract,  standard,  specification,  or  other  formally  imposed 
document  [lEE98d], 

330 

The  act  or  process  of  assigning  a  number  or  category  to  an  entity  to  describe  an  attribute  of  that  entity.  A  figure,  extent,  or  amount 

obtained  by  measuring  [IEE98d], 

331 

A  decision  aid  for  organizing,  selecting,  communicating,  and  evaluating  the  required  quality  attributes  for  a  software  system.  A 

hierachical  breakdown  of  quality  factors,  quality  subfactors,  and  metrics  for  a  software  system. 

332 

A  function  whose  inputs  are  software  data  and  whose  output  is  a  single  numerical  value  that  can  be  interpreted  as  the  degree  to  which 

software  possess  a  given  attribute  that  affects  its  quality  [IEE98d], 

333 

A  class  whose  instances  are  classes.  Metaclasses  are  normally  used  to  build  metamodels  [UML99], 

334 

A  metric  used  to  measure  the  characteristics  of  any  intermediate  or  final  product  of  the  software  development  process  [IEE98d] . 
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ARL  4  -  The  fourth  level  of  the  architecture  readiness  level,  the  SSA  Implementa¬ 
tion  Layer  is  established.  System  viewpoints  from  the  Domain  or  Enterprise  Metadata 
Registry  or  the  SSA  Component  Layer  defined  and  understood.  Views  and  requirements 
are  mature.  Architectural  components  establishing  the  view  have  validated  descriptive  and 
classification  infonnation.  Metadata  analysis  is  largely  complete  and  metadata  employed 
from  the  Metadata  Registry.  Product  line  analysis  is  underway.  Validation  data  availabil¬ 
ity  determined.  Metrics  framework  and  software  quality  metrics  process  detennined.  Core 
assets  determined.  Final  results  received  from  Domain  or  Enterprise  Metadata  Repository. 
The  SSA  is  mature  enough  to  start  software  engineering  planning  efforts. 

ARL  5  -  The  fifth  level  of  the  architecture  readiness  level  is  the  transition  point 
from  the  SSA  to  conceptual  model  development  in  the  software  engineering  process.  Ar¬ 
chitectural  assets  are  available  from  the  Domain  or  Enterprise  Metadata  Registry.  Product 
line  conceptual  model  development  initiated.  Validation  data  availability  confirmed.  Met¬ 
rics  framework  and  software  quality  metrics  transitioned  to  the  conceptual  framework. 
Conceptual  Model  Verification  and  Validation  Plan  developed.  Conceptual  model  devel¬ 
oped.  Conceptual  model  validated.  The  architecture  should  be  matured  to  at  least  ARL  5 
before  it  is  ready  for  software  engineering. 

ARL  6  -  The  sixth  level  of  the  architecture  readiness  level  is  the  system  analysis 
and  design  phase  of  the  software  engineering  process.  Interface  control  documents  are 
complete.  Architectural  assets  identified  from  the  Domain  or  Enterprise  Metadata  Registry 
for  component  or  system  development.  Product  line  conceptual  model  validated.  Vali¬ 
dated  data  for  validation  phase  is  available.  Metrics  framework  and  software  quality  met- 
rics  transitioned  to  the  development  phase,  and  metrics  validated  .  Implementation  Veri¬ 
fication  and  Validation  Plan  developed.  Analysis  and  design  activities  completed  and  veri¬ 
fied. 

ARL  7  -  The  seventh  level  of  the  architecture  readiness  level  is  the  component  and 
interface  development  phase  of  the  software  engineering  process.  ARL  7  is  optional  if 
component-based  development  methods  not  employed.  Interface  control  documents  are 
complete.  Architectural  assets  are  available  from  the  Domain  or  Enterprise  Metadata  Reg- 
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The  act  or  process  of  ensuring  that  a  metric  reliably  predicts  or  assess  a  quality  factor  [IEE98d]. 
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istry  for  component  development.  Product  line  component  model  validated.  Validated 
data  used  for  component  validation  phase.  Component-level  predictive  metrics336  estab¬ 
lished.  Metrics  framework  and  software  quality  metrics  transitioned  to  the  component- 
level  development  phase.  Implementation  Verification  and  Validation  Plan  developed. 
Analysis  and  design  activities  completed  and  verified.  Components  developed  and  tested. 

ARL  8  -  The  eighth  level  of  the  architecture  readiness  level  is  the  system  imple¬ 
mentation  phase  of  the  software  engineering  process.  System  interface  control  documents 
are  complete.  Architectural  assets  are  available  from  the  Domain  or  Enterprise  Metadata 
Registry  for  system  development.  Product  line  system  model  validated.  Validated  data 
used  for  system  validation  phase.  System-level  predictive  metrics  established.  Metrics 
framework  and  software  quality  metrics  transitioned  to  the  system-level  development 
phase.  Implementation  Verification  and  Validation  Plan  executed.  System-level  develop¬ 
ment,  test,  and  integration  activities  completed  and  verified.  System  transitioned  fonn  the 
software  engineering  phase  to  the  operation  and  support  phase. 

ARL  9  -  The  ninth  level  of  the  architecture  readiness  level  is  the  system  operations 
and  support  phase  of  the  software  life  cycle.  Product  line  system  model  variant  main¬ 
tained.  Validated  data  is  archived.  System-level  predictive  metrics  exercised.  Metrics 
framework  and  software  quality  metrics  monitored  and  analyzed.  System-level  develop¬ 
ment,  test,  and  integration  documentation  maintained  as  core  assets.  System  transitions 
from  the  operation  and  support  phase  into  the  retirement  phase. 

G.  DOMAIN  INTEGRATED  PRODUCT  DEVELOPMENT  TEAMS  (DIPDT) 

Implementing  software  product  lines  in  the  Department  may  be  even  more  chal¬ 
lenging  since  the  Department  acquires  software  system  as  opposed  to  developing  the  soft¬ 
ware.  If  credible  authoritative  model  representations  populating  credible  simulations  sup¬ 
porting  decision-makers  confidence  in  the  results  is  the  strategic  end  we  wish  to  achieve; 
and  the  Software  Architecture-Based  Product  Line  Model  is  the  way,  we  suggest  that  the 
Domain  Integrated  Product  Development  Team  (DIPDT),  employing  mature  Product  Line 
Practices  described  in  Chapter  VIII,  are  the  means  to  accomplish  this  strategic  objective. 

A  metric  applied  during  the  development  and  used  to  predict  the  values  of  a  software  quality  [!EE98d], 

320 


336 


Recall  in  earlier  chapters  we  discussed  contract  complexity.  In  Figure  10-17,  a  no¬ 
tional  Missile  Defense  Agency  organizational  chart  suggests  the  challenges  posed  by  sev¬ 
eral  layers  of  government  agencies  supporting  the  Missile  Defense  Agency.  Many  of  these 
agencies  in  turn  have  contractual  agreements  with  one  or  more  contractors  to  provide  ser¬ 
vices  or  deliverable  products.  The  contractors  in  turn  may  have  contractual  agreements 
with  one  or  more  sub-contractors.  The  contracts  may  differ  in  many  ways  such  as  scope, 
period  of  perfonnance,  deliverables,  contract  incentives,  and  other  contractual  clauses.  The 
communication  channels  also  vary  govemment-to-government,  government-to  contractor, 
contractor-to-contractor,  organization-to-organization,  and  even  within  the  respective  or¬ 
ganizations. 


Figure  10-17.  MDA  Contract  Complexity  (Notional) 

Figure  2-1  and  Table  7-1  identified  a  consistent  pattern  of  Department  institutional 
constraints  affecting  M&S  credibility.  The  reoccurring,  non-technical  constraints  such  as 
process,  culture,  organization,  and  management,  in  addition  to  the  more  technical  con- 
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straints  of  software  development,  architecture,  and  engineering,  suggest  several  pre¬ 
conditions  require  resolution  before  the  Department  may  realistically  plan  for  credible 
simulation  software  and  improved  confidence  in  the  results.  One  possible  means  to  im¬ 
prove  the  pre-conditions  identified  in  the  research  involves  the  development  of  a  Domain 
Integrated  Product  Development  Team  (DIPDT). 

The  Integrated  Process  Team  (IPT)  concept  is  not  new  to  the  Department,  and  many 
principles  of  the  IPT  process  support  the  proposed  DIPDT  concept.  The  DIPDT  concept 
illustrated  in  Table  10-2  also  integrates  tailored  versions  of  the  software  product  line  meth¬ 
odology  and  practice  line  areas,  cited  in  earlier  chapters.  The  DIPDT  concept  evolved  with 
the  research  as  it  became  readily  apparent  that  no  “silver  bullet”  approach  would  solve  the 
Department’s  systemic  problems  with  achieving  credible  authoritative  representations.  In¬ 
stead,  a  Domain-Managed,  de-centrally  executed  product  team  approach  evolved  from  the 
research  of  the  Agency  System-Level  Core  M&S. 
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Table  10-2.  Domain  Integrated  Product  Development  Team  (DIPDT) 

Table  10-2  illustrates  a  subset  of  a  notional  Agency  DIPDT  model,  showing  a  vir¬ 
tual  organization  formed  around  the  software  product  line.  In  this  case  the  software  prod¬ 
uct  lines  are  the  five  core  simulations  cited  in  Chapter  IX  (e.g.,  CAPS,  MDWAR,  EAD- 
SIM,  EADTB,  MDSE)  shown  as  Simulation  A  through  Simulation  E  on  the  vertical  axis  or 
partition.  On  the  horizontal  axis  of  the  array  a  partial  list  of  key  stakeholders  supporting 
the  software  product  line,  both  government  and  contractor  for  the  actual  systems  /  sub¬ 
systems  themselves  and  the  model  developers.  The  system  engineer,  developmental  tester 
[Eic94],  and  operational  tester  play  significant  roles.  Key  DIPDT  members  also  include 
the  V&V  agents  and  the  accreditation  agents. 

Note  line  9  through  line  15  of  Table  10-2.  In  this  view  of  the  notional  VIDPT, 
Element  A  is  further  decomposed  into  the  system  and  sub-system  levels  with  representa¬ 
tives  on  each  simulation  DIPDT  in  which  their  system  or  sub-system  is  represented.  Con¬ 
tracting  Officer  Representatives  (COR)  or  Contracting  Officer  Technical  Representatives 
(COTR)  play  a  key  role  on  the  DIPDT  as  matrix  support  for  contract  management  support. 
The  objective  of  the  DIPDT  is  to  develop  a  domain  integrated  product  team  approach  based 
on  the  primary  source  level  of  the  authoritative  model  representation  infonnation  through 

the  various  levels  of  reoccurring,  non-technical  constraints  to  the  simulation  product  line 
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level.  The  simulation  product  line  in  Table  10-2  would  replace  the  traditional  tree  structure 
in  Figure  10-17.  We  envision  that  as  the  DIPDT  and  the  SSA  concept  mature,  and  meta¬ 
data  repositories  appear,  a  wider  variety  of  credible  and  architecturally  sound  model  repre¬ 
sentations  will  be  available  to  support  the  future  development  of  composable  simulations. 

H.  SUMMARY 

Chapter  X  introduced  the  detailed  research  design  for  the  Simulation  Software  Ar¬ 
chitecture-Based  Product  Line  Model  including  five  major  elements  supporting  the  method, 
specification,  design,  and  implementation.  The  chapter  provided  an  architectural  analysis 
of  the  Department’s  existing  de  facto  M&S  software  architecture  and  introduced  the  Simu¬ 
lation  Software  Architecture  (SSA),  the  Simulation  Software  Architectural  Framework 
(SSAF),  the  Simulation  Product  Lines  Architecture  (SPLA),  the  Simulation  Software  Ar¬ 
chitecture-Based  Product  Line  Model  Domain  Metadata  Repository,  and  Architecture 
Readiness  Levels  (ARL). 

The  Simulation  Software  Architecture  (SSA)-based  Model  is  an  abstract  software 
architecture-based  horizontal  foundation  supporting  multiple  viewpoints  and  views.  The 

'X'X'l 

Simulation  Software  Architectural  Framework  (SSAF)  is  a  second  vertical-slice  architec¬ 
tural  component  overlaying  the  SSA,  which  includes  system  and  environment  components. 
The  Simulation  Product  Lines  Architecture  (SPLA)  provides  the  framework  to  manage  the 
variability338  [BB00,  BosOO,  DMH01,  RSC00,  MNJ+02,  CBB+03],  features  [BosOO, 
KLD02],  and  differences  between  products  and  comprises  the  third  element. 

The  Simulation  Software  Architecture-Based  Product  Line  Model  Domain  Meta¬ 
data  Repository  provides  the  structure  for  the  Domain  Metadata  Registry  modeled  from 
[IS079c]  to  ensure  interoperability  with  Department,  Federal  Government,  and  private  sec¬ 
tor  metadata  registries.  The  SSA  and  SSAF  establish  the  architectural  foundation  for  the 
SPLA,  which  supports  variability  and  extensibility  of  the  architectural  construct.  We  also 
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In  UML  this  is  called  a  partition  or  a  set  of  related  classifiers  or  packages  at  the  same  level  of  abstraction  or  across  layers  in  a  layered 
architecture  [UML99], 

338  Variability  refers  to  decisions  that  will  be  made  by  a  member  of  the  development  team  prior  to  system  deployment  [CBB+03]. 
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adapted  the  architectural  description  conceptual  framework339  from  [IEE00B]  into  the  SSA, 
SSAF,  and  SPLA  models  composing  the  Simulation  Software  Architecture -Based  Product 
Line  Model. 

Core  asset  development  is  an  evolving  practice.  Mining  existing  assets,  a  core  asset 
development  option,  offers  an  organization  the  potential  to  leverage  all,  or  part,  of  its  cu¬ 
mulative  system  investments,  and  represents  a  critical  practice  area  supporting  a  software 
product  line  initiative.  However,  there  are  significant  risks  achieving  success  developing 
software  product  lines.  Some  reasons  include  the  high  risks  due  to  the  lack  of  documenta¬ 
tion,  the  poorly  maintained  state  of  many  existing  systems,  and  the  fact  that  many  systems, 
initially  developed  for  different  purposes,  lack  support  for  current  software  engineering  ap¬ 
proaches.  Research  indicates  this  is  generally  true  in  the  Department’s  M&S  domain. 

The  chapter  provides  an  overview  of  a  reverse-engineering  and  reengineering  envi¬ 
ronment  framework  to  improve  program  understanding  to  support  software  architecture 
reconstruction  architecture  for  model  representations.  Reengineering  has  the  potential  to 
improve  an  organization’s  understanding  of  the  software  and  improve  it  and  must  improve 
on  past  failures.  In  many  existing  software  systems,  the  developers  conceived,  designed, 
and  developed  projects  as  unique  products,  with  minimal  integration,  and  very  little  sys¬ 
tematic  reuse  of  assets,  which  lose  value  over  time  by  getting  stale  and  requiring  more  as¬ 
sets  to  maintain  them. 

Software  architecture  reconstruction  provides  a  mechanism  for  the  effective  reuse 
of  software  assets.  Architecture  reuse  is  the  cornerstone  practice  of  product  line  develop¬ 
ment,  supported  by  three  sources  for  architecture  recovery: 

•  Source  code,  which  is  authoritative,  but  does  not  hold  all  the  information, 

•  Documentation,  which  is  normally  undependable,  incomplete,  or  outdated, 

•  Human  experts,  who  may  be  dependable,  but  biased  [CW98]. 

Architecture  recovery  and  reconstruction  is  an  iterative  and  interactive  labor- 
intensive  effort,  dependent  upon  the  level  of  specification,  documentation,  dissemination 
and  control  of  the  legacy  architecture  artifacts.  Architecture  recovery  methods  vary  from 


339 

The  conceptual  framework  is  the  term  of  reference  for  the  architectural  description  and  establishes  the  terms  and  concepts  for  the 
content  and  use  of  architectural  descriptions  [lEEOOb], 
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entirely  manual  reconstruction  processes  to  tool-supported  manual  reconstruction  and  semi- 
autonomous  reconstruction  techniques,  and  may  include  data  mining  and  employment  of 
architecture  description  languages.  Pattern  analysis  supports  architecture  recovery  and  re¬ 
construction  efforts. 

The  Architecture  Readiness  Levels  (ARL)  supports  future  architectural  components 
and  products  developed  from  this  methodology,  is  the  fifth  and  final  contribution.  The 
chapter  also  suggests  a  complementary  Domain  Integrated  Product  Development  Team 
(DIPDT)  approach  for  implementing  the  Simulation  Software  Architecture-Based  Product 
Line  Model,  to  reduce  the  cycle  time  for  resolving  the  multiple  dimensions  causing  hetero¬ 
geneous  system  representation  anomalies,  and  improving  federation  interoperability. 
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XI.  CONCLUSIONS  AND  RECOMMENDATIONS  FOR  FUTURE  WORK 


A.  REVIEW  OF  MAJOR  RESEARCH  AREAS  RELEVANT  TO  THIS  WORK 

This  research  focuses  on  the  major  research  areas  introduced  in  Figure  2-1,  and  rep¬ 
licated  in  Figure  11-1  for  improving  the  current  heterogeneous  system  representation 
anomalies-based  issues,  which  erodes  credibility  in  the  Department  simulation  process,  and 
injects  doubt  instead  of  confidence  in  the  simulation  results  supporting  Departmental-  and 
National-level  decision-makers. 


Figure  11-1.  Major  Areas  Researched  for  this  Work 

In  Chapter  II  we  identified  the  seven  major  areas  affecting  improved  credibility  in 
simulation  model  representations  supporting  confidence  in  the  Department-  and  National 
Level  decision-making  process: 

•  An  Architectural  Framework, 

•  Software  Engineering, 

•  Process  Improvement, 
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•  Quality, 

•  Verification  and  Validation, 

•  Heterogeneous  Data, 

•  Technical  and  Substantive  Interoperability. 

Five  of  the  seven  areas  affecting  simulation  credibility  were  methods  to  improve  the 
simulation  product,  and  two  of  the  seven  areas  (e.g.,  heterogeneous  data,  technical  and  sub¬ 
stantive  interoperability)  were  conditions  adversely  affecting  model  credibility.  While  all 
of  the  five  simulation  product  improvement  methods  experienced  some  progress,  they  also 
encountered  several  limitations: 

•  The  Department’s  M&S  architectural  framework  remains  immature.  The  Common 
Technical  Framework  including  the  High-Level  Architecture,  Conceptual  Model  of 
the  Mission  Space,  and  improved  data  quality  initiatives  did  not  significantly  im¬ 
prove  simulation  model  representation  credibility.  The  High-Level  Architecture 
achieved  technical  success,  and  the  IEEE  adopted  it.  However,  it  is  too  early  to  tell 
whether  the  HLA  will  be  a  commercial  success,  now  that  the  Department  no  longer 
financially  underwrites  it.  The  Conceptual  Model  of  the  Mission  Space  model  im¬ 
plementation  proved  inconsistent,  reducing  credibility  in  the  V&V  process.  Data 
remains  a  major  problem  [DoDOla],  The  current  object  models  (e.g.,  SOM,  FOM, 
BOM)  do  not  address  the  substantive  interoperability  issue.  In  addition  the  JTA 
and  the  HLA  remain  incompatible. 

•  The  Department’s  software  engineering  strategy  to  replace  many  large-scale  legacy 
simulations  with  new,  well-engineered,  joint  simulations  (e.g.,  JMASS,  JSIMS, 
JWAR)  experienced  significant  cost  overruns,  failed  to  meet  continually  extended 
delivery  schedules,  and  met  only  a  subset  of  the  approved  requirements.  With  ma¬ 
jor  funding  reductions  or  planned  cancellation,  the  future  of  the  three  major  simula¬ 
tion  software  engineering  initiatives  appear  limited  at  best.  As  a  result  the  Depart¬ 
ment  is  reviewing  options  to  extend  the  life  span  of  the  existing  large-scale,  legacy 
simulation  systems. 

•  Although  process  improvement  in  no  way  guarantees  fewer  errors  in  the  final  soft¬ 
ware  product,  initial  indications  over  the  past  decade  suggest  that  properly  disci¬ 
plined  process  improvement  has  a  favorable  impact  on  software  development  qual¬ 
ity  and  may  improve  simulation  model  credibility.  Several  process  methodologies 
exist  including  the  CMM,  CMMI,  Software  Acquisition  Management,  and  the 
Product  Line  Practice  Areas.  Our  research  indicates  that  mature  processes  are  pre¬ 
conditions  for  the  development  of  credible  simulation  model  representations. 

•  Quality  remains  an  elusive  element  in  the  Department’s  M&S  portfolio.  In  many 
cases  the  discussion  of  M&S  quality  eventually  lead  to  debates  about  fidelity.  Fi¬ 
delity  as  a  term  has  many  appropriate  uses,  but  some  suggest  that  fidelity  is  an 
overused  tenn,  open  to  almost  any  meaning.  Our  research  indicates  that  quality  at¬ 
tributes  are  also  pre-conditions  for  simulation  credibility,  and  are  key  architectural 
components  of  the  proposed  Simulation  Software  Architecture  Model. 
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•  The  Department’s  V&V  process  is  the  primary  method  for  establishing  simulation 
credibility.  However,  systemic  issues  with  incomplete  or  non-existent  conceptual 
models,  a  general  lack  of  validation  data,  and  inconsistent  V&V  implementation 
raises  many  questions  of  a  simulation’s  credibility.  The  accreditation  process,  de¬ 
pendent  upon  a  sound  V&V  process,  lacks  credibility  as  a  result. 

The  research  results  presented  earlier  in  Chapters  II  and  IX  indicates  that  after  fifty 
years  of  development  the  Department  M&S  domain  has  not  achieved  the  desired  level  of 
credibility  and  confidence  needed  to  support  the  Department  decision-making  process,  and 
existing  methods,  techniques,  and  procedures  have  not  achieved  the  desired  objectives. 

A  major  finding  of  this  research  suggest  that  there  is  little  supporting  evidence  that 
the  Department  M&S  software  development  community  will  achieve  future  success:  1)  im¬ 
proving  heterogeneous  system  representation  anomalies,  2)  improving  overall  simulation 
credibility;  and  3)  improving  confidence  in  results  provided  by  Department  M&S  employ¬ 
ing  current  conceptual  model  development  paradigms.  In  addition,  many  of  the  past  meth¬ 
ods  and  recommendations  to  improve  the  Department’s  M&S  portfolio  have  been  reiterated 
and  reissued,  and  it  is  uncertain  if  more  time  and  more  resources  will  result  in: 

•  Better  verification  and  validation  (V&V)  techniques, 

•  Better  discipline  in  the  software  development  process, 

•  Improved  accreditation  processes, 

•  Improved  use  of  data, 

•  Improved  quality, 

•  Improved  documentation, 

•  Improved  coordination  between  the  users  and  the  developers, 

•  Improved  credibility, 

•  Improved  confidence  in  the  results. 

B.  EVALUATION  OF  A  SIMULATION  SOFTWARE  ARCHITECTURE 
METHOD  AGAINST  THE  CURRENT  M&S  TECHNIQUES 

It  is  clear  from  the  research  that  Department  and  the  Service  Components  acted 
with  due  diligence  to  improve  the  quality  of  Department  M&S.  By  the  mid-1990s  the  De¬ 
partment  established  M&S  management  organizations,  developed  policies,  implemented 
plans,  allocated  significant  funding,  and  executed  major  new  programs  with  the  goals  and 
objectives  of  improving  Department  M&S  practices.  However,  not  withstanding  these 
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considerable  efforts,  major  concerns  still  exist  about  the  credibility  of  the  Department 
simulation  process.  There  are  several  major  reasons  for  this  lack  of  credibility  in  Depart¬ 
ment  M&S. 

The  first  contributor  was  the  software  industry-wide  hope  for  a  “silver  bullet”  solu¬ 
tion  to  resolve  the  Department  “software  crisis”.  However,  as  hope  for  a  “silver  bullet” 
technical  breakthrough  to  improve  Department  software,  including  simulation  software, 
waned  in  the  1990s,  it  became  apparent  that  other  non-technical  solutions  were  required. 

In  addition,  new  Department  M&S  management  organizations,  such  as  the  DMSO,  had  to 
overcome  Service  parochialism,  the  unpredictability  of  the  Department  budget  process,  the 
vagaries  of  the  Department  acquisition  process,  and  over  forty  years  of  unplanned  Depart¬ 
ment  M&S  growth.  Then  in  the  late  1990s  significant  funding  and  manpower  resources 
originally  programmed  to  improve  Department  M&S  were  committed  to  the  Year  2000 
problem  resolution  project. 

A  second  underlying  cause  for  the  current  situation  is  the  Department  institutional 
constraints  affecting  M&S  credibility,  and  the  apparent  slow  maturity  progress  of  major 
M&S  credibility  factors  cited  in  Table  7-1.  In  1996  a  major  Department  M&S  study 
[PHP+96]  identified  three  major  types  of  systemic  Department  M&S  challenges,  technical, 
cultural,  and  managerial;  while  concurrently  the  Simulation-Based  Acquisition  (SBA)  ini¬ 
tiative  identified  three  similar  factors,  process,  culture,  and  environment  as  SBA  enablers. 
More  recently  the  SEI  endorsed  three  related  practice  areas,  technical  management,  and 
organizational,  to  support  the  successful  implementation  of  software  product  lines. 

This  is  significant  since  these  factors  represent  an  institutional  realization  that  the 
resolution  to  the  twenty  year  old  “software  crisis”  is  not  a  “silver  bullet”  solution.  Instead, 
the  problem  and  the  solution  are  multi-dimensioned,  complex  issues  that  require  changes  in 
many  areas.  This  change  will  take  time  for  the  Department  to  truly  assimilate  and  adapt  to 
during  the  transition  phase  into  the  Infonnation  Age.  We  must  however,  also  be  mindful  of 
the  many  Industrial  Age  vignettes  where  nations,  armies,  navies,  or  air  forces  were  slow  to 
adapt  and  the  severe  consequences  of  that  that  failure  in  terms  of  defeat  or  loss  of  life. 

Software  quality,  a  third  contributing  study  factor,  continues  to  lag  well  behind  the 
computer  industry’s  hardware  engineering  segment  progress  on  quality,  cost,  and  perform  - 
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ance.  While  computer  hardware  engineering  is  well  into  the  fourth  computer  era  identified 
by  [Pre97]  and  employs  the  production  approach  to  quality  according  to  criteria  established 
by  [Wal94],  simulation  software  quality  is  best  labeled  as  transcendental.  In  contrast  to 
the  mature  computer  hardware-engineering  sector,  the  software  engineering  sector  of  the 
computer  industry  is  just  now  achieving  a  consensus  on  a  software  engineering  body  of 
knowledge  and  agreement  on  the  benefits  of  software  process  improvement.  The  imple¬ 
mentation  of  Department  M&S  quality  objectives  for  improving  quality  attributes,  includ¬ 
ing  fidelity,  accuracy,  resolution,  error,  detail,  aggregation/disaggregation,  and  multiresolu¬ 
tion  modeling  techniques  have  achieved  only  limited  success. 

Software  architecture  is  the  fourth  study  factor  acknowledged  to  strongly  influence 
a  system  over  its  life  cycle.  While  computer  hardware-related  architectures  supported  by 
Moore’s  Law  were  ascendant  over  the  past  fifty  years,  software-related  architectural  con¬ 
siderations,  if  they  existed,  were  of  secondary  importance  to  the  overall  system  engineering 
and  development  process.  Today,  the  cost  and  complexity  of  software-intensive  systems 
have  changed  the  industry’s  dynamics  and  the  publication  of  software  architectural  descrip¬ 
tions  [IEEOOb]  and  styles  [SC96,  BK99,  CBB+03]  for  software-intensive  projects  are  in¬ 
troducing  software  architectural  principles  to  the  emerging  software  engineering  commu¬ 
nity.  However,  the  Department’s  life  cycle  management  of  software-intensive  simulation 
systems  lacks  software  architecture  concepts  beyond  the  application  of  the  CTF  (e.g.,  HLA, 
Conceptual  model,  data). 

The  Department  legacy  M&S  portfolio  is  the  fifth  major  study  factor.  The  Depart¬ 
ment  developed  a  significant  and  expensive  portfolio  of  legacy  M&S  over  the  past  fifty 
years.  Although  an  operational  system  has  a  de  facto  architecture,  many  of  these  M&S 
evolved  in  an  ad  hoc  manner  and  have  become  large  legacy  systems  with  multiple  stake¬ 
holders  and  expensive  support  infrastructures.  A  significant  number  of  these  systems  lack 
conceptual  models  and  have  an  ad  hoc  V&V  history.  In  some  cases  the  developer  created 
conceptual  models  after  the  fact,  if  at  all,  and  adherence  to  disciplined  M&S  V&V  process 
was  a  low  priority. 

Department  agencies  also  have  a  tendency  to  use  these  systems  for  studies  or  tests 
for  which  they  are  ill  suited,  in  order  to  show  a  return  on  investment.  As  a  result  the  ac- 
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ceptability  criteria  and  caveats  developed  by  the  accreditation  process  for  these  simulations 
may  be  of  limited  value  or  counter-productive  to  the  original  purpose  and  intent  of  the 
study  or  test  objectives.  Complicating  the  current  situation,  replacement  systems  are  over¬ 
due.  The  new  Joint  M&S  programs  (e.g.,  JWARS,  JSIMS,  JMASS)  scheduled  to  replace 
many  legacy  systems  are  all  late,  over  cost,  and  reducing  the  original  requirements  to  meet 
a  future  operational  status. 

M&S  verification  and  validation,  a  sixth  study  factor,  is  the  foundation  for  M&S 
credibility  and  confidence.  However,  many  consider  V  &  V  expensive  and  time  consum¬ 
ing.  In  this  research  we  reviewed  the  V&V  process  in  the  context  of  supporting  Depart- 
ment-and  National-level  decisions  with  improved  credibility  and  confidence,  and  con- 
finned  previous  assessments  indicating  the  quality  of  Department  V&V  practices  are  ad 
hoc  and  inconsistently  applied. 

The  conceptual  model  of  the  mission  space  (e.g.,  CMMS  or  FDMS),  a  seventh 
study  factor,  is  the  approved  foundation  for  a  rigorous  Department  verification  and  valida¬ 
tion  program.  The  Department  bases  successful  V  &  V  on  a  verified  and  validated  concep¬ 
tual  model.  In  practice,  conceptual  model  development  and  conceptual  model  validation  as 
a  precursor  to  verifying  the  software  implementation,  rarely  occurs.  The  M&S  community 
developed  many  conceptual  model  formats  [RPGOO],  however,  there  is  no  consensus  on  a 
standard  conceptual  model  at  this  time. 

The  current  Department  M&S  Hierarchy,  the  eighth  study  factor,  provides  only 
general,  qualified  levels  for  M&S  fidelity  and  resolution,  provides  only  limited  support  for 
the  user  to  ascertain  a  simulation’s  attributes.  The  model  provides  little  if  any  support  for 
quantifiable  measures  of  simulation  quality.  In  addition,  metrics  based  on  this  model  tend 
to  be  qualified,  subjective,  and  lack  authority.  In  this  research  we  analyzed  the  current  De¬ 
partment  M&S  Hierarchical  model  as  a  core  asset  and  reviewed  alternatives  for  mining  re¬ 
verse  engineering  and  reengineering  these  core  assets  into  product  line  architecture. 

Common  limitations  of  the  Department’s  efforts  to  improve  M&S  credibility  and 
confidence  in  results  include  the  following  reasons.  First,  a  significant  amount  of  current 
Department  M&S  systems  were  developed  in  previous  decades  and  are  currently  in  a  main- 
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tenance  mode,  suggest  a  strategy  of  introducing  improvements  in  a  planned  evolutionary 
manner. 

Second,  the  M&S  policies,  procedures,  best  practices,  and  funded  projects  devel¬ 
oped  in  the  mid-1990s  may  need  more  time  to  promulgate  and  affect  M&S  development 
processes  for  new  models  and  simulations. 

Third,  in  the  1990s,  the  Department  experienced  a  major  shift  from  M&S  support¬ 
ing  a  bi-polar  world  with  two  super  powers  to  a  world  characterized  by  asymmetrical  war¬ 
fare  and  weapons  of  mass  destruction,  and  support  Department  transfonnation  initiatives,  at 
a  pace  of  change  seldom  experienced  in  the  Department. 

Fourth,  as  the  Department  evolves  from  a  threat-based,  requirements-driven  organi¬ 
zation  and  transforms  itself  to  a  capability-based  model  while  concurrently  maintaining  the 
ability  to  execute  the  National  Military  Strategy,  the  question  remains,  are  we  developing 
the  right  M&S  (e.g.,  are  we  building  the  right  thing)? 

Last,  if  we  detennine  that  we  are  building  the  right  thing,  have  the  software  engi¬ 
neering  capabilities  of  the  Department  matured  enough  to  “build  it  right?”  The  product 
line  domain  architecture  supporting  specified  requirements  allocated  from  the  system  engi¬ 
neering  process  presented  in  Chapter  X  of  this  dissertation  address  these  limitations. 

Accomplishing  this  significant  undertaking  requires  a  mature  population  of  soft¬ 
ware  engineers  and  software  architects.  Grounded  in  the  technical  foundations  of  the  field 
(e.g.,  computer  science,  engineering,  infonnation  technology),  and  the  emerging  software 
architecture  discipline,  future  software  engineers  and  software  architects  will  need  to  un¬ 
derstand  the  viewpoints  of  multiple,  diverse  stakeholders  at  all  levels  of  the  Enterprise  and 
synchronize  composable  software  engineering  solutions  that  meet  the  cost,  perfonnance, 
and  schedule  constraints. 

Distributed,  integrated  product  development  teams  software  engineers  and  software 
architects  working  with  authoritative  product  line  repositories  of  valid  domain  data  provide 
one  such  method  for  improving  future  simulation  credibility.  Software  engineering  and 
software  architectures  are  now  emerging  as  critical  domain  disciplines  that  must  continue 
to  mature  the  corresponding  bodies  of  knowledge,  if  the  Department  and  the  Nation  are  to 
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exploit  the  full  potential  of  computer  technology,  and  set  the  foundation  for  the  next  great 
era  in  computing,  the  Software  Architecture  Era. 

C.  RECOMMENDATIONS  FOR  FUTURE  RESEARCH 

The  simulation  software  architecture-based  product  line  approach  to  simulation 
credibility,  supported  by  architecture  readiness  levels  and  domain  integrated  product  devel¬ 
opment  teams  provides  five  proposed  areas  for  future  research.  The  Simulation  Software 
Architecture  (SSA)-Based  Model  provides  an  abstract  software  architecture -based  horizon¬ 
tal  foundation  supporting  multiple  viewpoints  and  views.  Second,  the  Simulation  Software 
Architectural  Framework  (SSAF)  adds  a  second  vertical-slice  architectural  component 
overlaying  the  SSA,  which  includes  system  and  environment  components. 

These  architectural  artifacts  support  the  Simulation  Product  Lines  Architecture 
(SPLA)  providing  the  framework  to  manage  the  variability,  features  and  differences  be¬ 
tween  products.  The  fourth  component,  the  Simulation  Software  Architecture-Based  Prod¬ 
uct  Line  Model  Domain  Metadata  Repository  provides  the  structure  for  the  Domain  Meta¬ 
data  Registry  to  ensure  interoperability  with  Department,  Federal  Government,  and  private 
sector  metadata  registries.  The  Architecture  Readiness  Levels  (ARL)  to  measure  future 
architectural  components  and  products  developed  from  this  methodology  is  the  fifth  area 
suggested  for  future  research. 

1.  Extension  of  the  Simulation  Software  Architecture  (SSA)-Based  Model 
into  Different  Domains 

The  lack  of  a  true  software  architecture  foundation  for  the  Department’s  simulation 
system  development  is  the  equivalent  of  being  on  the  road  to  Abilene340  without  a  plan,  and 
nobody  knows  why  we  are  going.  The  SSA  Model  complements  and  supports  the  existing 
HLA,  with  a  focus  on  improving  the  conceptual  models  and  data  elements  of  the  current 
Common  Technical  Framework.  Figure  1 1-2  illustrates  the  current  plan  to  implement  the 
Missile  Domain  SSA.  The  first  phase  involves  decomposing  the  domain  M&S  hierarchy. 
The  second  phase  involves  the  identification  of  patterns  for  future  product  lines.  Chapter 
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Defense  Systems  Management  College  teaching  point  in  the  Program  Management  Course  reinforcing  the  need  for  a  program  plan. 
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IX  provided  the  pattern  analysis  for  the  first  two  phases.  The  final  phase  involves  the  core 
asset  mining  and  architecture  reconstruction  for  the  SSA. 

The  development  of  views  and  viewpoints  suggest  the  need  for  a  broad-based  re¬ 
search  effort  in  multiple  domains  and  the  development  of  the  domain  metadata  repository 
to  support  the  development  of  simulation  software  architecture  description  (SimSAD). 


Horizontal  Integration 


1  Lacks  Planned  Architecture 
Defining  Layers 
1  Simulations  Are  Horizontally 
Managed 


■  Adds  Vertical  Dimension 

■  Supports  Benchmarking  (Anchoring) 

■  Develop  Element  Conceptual  Model 
For  BMDS  At  AU  Levels  Of  The 
Hierarchy 

■  Supports  Improved  V&V 


Four  -Layer  Software  Architecture* 

'  Referent.  Conceptual  Component,  System 
'  Domain  -Specific  (Missile  Defense)  Product  Lines 

■  Supports  Multiple  Views  Of  The  Missile  Defense 

■  Multiple  Components  “Views"  At  Each  Layer  Provide  Flexibility 

1  The  Implementation  At  The  Next  Higher  Level  Is  Allowed  To 

Use  Services  Of  Next  Lower  Level 

Relations  Between  Layers 

1  The  Sendees  In  The  Lower  Level  A 
The  Next  Higher  Level 


Allowed  To  Be  Used  By 


Figure  11-2.  Missile  Defense  Domain  SSA  Status  (After  [Gre03b]) 

2.  Simulation  Software  Architectural  Framework 

We  identified  the  SSAF  supporting  the  Simulation  System  Architectural  Descrip¬ 
tion  (SimSAD),  as  the  system’s  environment,  which  includes  the  physical  world  and  the 
external  objects,  conditions,  or  processes  influencing  the  behavior  of  the  system  in  past, 
present,  and  future  dimensions,  real  or  imagined,  but  clearly  defined  and  documented  in  the 
SimSAD.  The  SimSAD  provides  the  foundation  for  the  follow-on  SPLA  to  manage  the 
product  line  variability  of  a  SimSAD. 
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In  the  missile  defense  domain,  future  research  would  determine  if  the  SimSAD  pro¬ 
vided  the  scalability  to  manage  the  large  number  of  simulation  models  projected  to  support 
a  dynamic  block  evolution  of  the  missile  defense  system  in  addition  to  unplanned  experi¬ 
ments  and  excursions.  Future  research  on  the  environment  component  of  the  SSAF  could 
provide  insights  into  the  viability  of  this  approach  in  supporting  a  major  weapon  system 
acquisition,  which  must  function  perfectly  the  first  time  in  a  worldwide  environment. 

3.  Simulation  Product  line  Architecture 

The  SPLA  provides  the  architectural  capability  to  exploit  commonality.  There  is  a 
potential  requirement  for  thousands  of  authoritative  simulation  model  representations.  In 
addition,  the  BMDA  capability-based  acquisition  process  may  add  new  elements,  while 
tenninating  or  reducing  other  element  development.  This  scenario  indicates  an  additional 
need  to  maintain  accurate  configuration  management  of  variant,  version,  feature,  and  op¬ 
tion  requirements  of  past  and  future  unknown  systems.  New  system  developments  may 
also  add  more  complexity  since  the  systems  may  be  unprecedented.  Additional  research 
could  identify  the  boundaries  for  the  SPLA. 

4.  SSA  Domain  Metadata  Repository  Model 

The  SSA  Domain  Metadata  Repository,  when  fully  populated  will  maintain  a  Do¬ 
main  Metadata  Registry,  a  Domain  Metadata  Catalog  containing  instances  of  metadata  as¬ 
sociated  with  domain  data  resources,  supporting  the  use  of  search  portals  and  queries;  a 
Domain  MetaClass  Catalog,  which  supports  the  development  of  metamodels  with  a  class 
whose  instances  are  classes;  A  Metalanguage  Catalog  which  specifies  some  or  all  of  the 
aspects  of  a  MetaLanguage  used  in  the  Domain;  a  Domain  Meta-Metamodel  Catalog  or 
model  that  defines  the  language  for  expressing  a  metamodel;  a  Domain  MetaModel  Cata¬ 
log  defining  the  language  for  expressing  a  mode;  a  Domain  Ontologies  Catalog  including 
data  categorization  schemes,  thesauruses,  glossaries,  key-word  lists,  and  taxonomies;  a 
Domain  Schemas  Catalog  representing  database  tables  and  relationship  structures,  XML 
document  type  definitions  (DTD),  and  XML  schema;  Features,  versions,  variants,  options, 
and  quality  attributes,  with  associated  profiles;  and  Architecture  Readiness  Levels  (ARLs). 
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Additional  research  may  evaluate  if  data  is  visible,  available,  and  usable  when  needed  and 
where  needed  to  speed  decision-making,  including  the  DoD  Metadata  Registry  identified  in 
Figure  11-3. 
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Figure  11-3.  Future  Areas  of  Research  (From  [Ste03b]) 


5.  Architecture  Readiness  Levels 

Architecture  Readiness  Levels  provide  the  opportunity  to  anchor  simulation  credi¬ 
bility  and  decision-makers’  confidence  in  an  architectural  foundation  with  quality  attrib¬ 
utes.  Additional  research  in  improving  the  V&V  process  based  on  the  architectural  matur¬ 
ity  of  the  model  representations  has  potential  to  improve  decision-makers’  confidence  in 
future  simulation  results. 
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D.  CONCLUDING  REMARKS 

Chapter  I  posed  the  following  question:  How  may  the  Agency  better  define,  de¬ 
velop,  evolve,  manage,  and  maintain  authoritative  model  representations  in  the  five  se¬ 
lected  large-scale,  software-intensive  legacy  simulations  and  effectively  address  simulation 
interoperability  issues  (e.g.,  heterogeneous  system  representation  anomalies)  within  exist¬ 
ing  constraints  and  pre-conditions  (e.g.,  technical,  organizational,  managerial)  affecting 
credibility  in  the  five  selected  simulations  systems? 

The  following  hypothesis  was  posed  in  response  to  this  question:  Employing  a  do¬ 
main-managed,  four-layered  simulation  software  architecture -based  product  line  model 
with  a  referent  layer,  developed  in  a  distributed  domain  integrated  software  engineering 
environment  supporting  the  evolution  of  five  selected  Agency  legacy  simulations  can  im¬ 
prove  the  authority  of  representations  in  the  five  selected  missile  defense  simulations.  The 
Software  Architecture-Based  Product  Line  Model  developed  in  a  distributed  development 
environment  by  a  Domain  Integrated  Product  Development  Team  employing  Architecture 
Readiness  Levels  (ARL)  to  measure  quality  provides  such  a  methodology.  Hypothesis  test¬ 
ing  included  concurrent  implementation  of  the  techniques  and  procedures  in  the  Agency’s 
M&S  domain.  The  Software  Architecture-Based  Product  Line  Model  for  Simulation 
Model  representations  provides  such  a  methodology. 


338 


APPENDIX  A  -  BALLISTIC  MISSILE  DELENSE  ORGANIZATION  (1996-2001) 
A.  INTRODUCTION 

During  the  1996-2001  period,  the  Ballistic  Missile  Defense  Organization  (BMDO) 
consisted  of  two  major  components,  a  Theater-based  missile  defense  program  and  the  Na¬ 
tion’s  National  Missile  Defense  program.  Figure  A-l  identifies  the  major  U.S.  missile  de¬ 
fense  acquisition  programs  existing  in  2001,  including  international  collaborative  efforts, 
major  technology  programs,  and  the  supporting  BM/C2,  weapon,  sensor,  and  communica¬ 
tion  components.  Figure  A-l  also  identifies  the  service  or  organization(s)  responsible  for 
the  management  of  the  program  development  for  each  system  [Bau02], 


BMDO  BMDO 


Figure  A- 1 .  Missile  Defense  Organization  -  200 1  -  (After  [BMDOOa]) 


B.  THEATER  MISSILE  DEFENSE  (TMD)  ORGANIZATION  -  (1996-2001) 

No  single  system  can  possibly  protect  the  diverse  and  complex  threat  posed  by  en¬ 
emy  ballistic  missiles,  aircraft,  and  cruise  missiles  against  U.S.  and  coalition  forces,  se¬ 
lected  assets  and  population  areas  within  a  theater  of  operations.  The  TMD  and  supporting 
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the  Joint  Theater  Air  and  Missile  Defense  (JTAMD)  doctrine  plans  to  detect,  classify,  in¬ 
tercept,  and  destroy/negate  enemy  missiles  prior  to  launch,  or  while  in  flight.  The  BMDO, 
the  Joint  Theater  Air  and  Missile  Defense  Organization  (JTAMDO),  Combatant  Com¬ 
manders,  and  other  Department  Components  developed  a  Family  of  Systems  (FOS) 
[BMDOOb]  concept  to  provide  a  coherent,  cohesive,  and  effective  protection  capability. 

The  FOS  incorporated  four  major  pillars:  Attack  Operations341,  Active  Defense  of  post- 
launched  theater  missiles,  Passive  Defense342  through  early  warning,  and  an  integrated  Bat¬ 
tle  Management,  Command  and  Control,  Communications,  Computers,  and  Intelligence 
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(BMC4I)  to  provide  missile  defense  in-depth.  Figure  A-2  illustrates  the  complex  enemy 
threat  and  the  JTAMD  Family  of  Systems  developed  to  counter  them. 


THREAT  TYPES 


Figure  A-2. 


Family  of  Systems  (After  [BMDOOa,  BMDOOb]) 


Attack  operations  includes  both  pre-  and  post-launch  attack  of  enemy  transporter-  erector-  launchers  (TELs)  and  supporting  infra¬ 
structure  [BMDOOa], 

342 

Passive  defensive  measures  include  actions  such  as  deception,  dispersion,  or  protective  construction  to  minimize  the  enemy  ballistic 

missile  defense  effectiveness  [BMDOOa], 

343 

BMC41  includes  the  controlling  and  coordinating  procedures  and  systems  to  integrate  an  effective  defense  [BMDOOa], 
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1. 


PATRIOT  -  (1996-2001) 


The  Army’s  Patriot  missile  defense  system,  managed  by  the  Anny,  has  evolved 
since  1965,  first  as  the  Surface  to  Air  Missile  Development  (SAM-D),  followed  by  Pre- 
Planned  Product  Improvements  into  the  Patriot  Anti-Tactical  Missile  -  1  (PAC-1)  and  -2 
(PAC-2)  systems.  In  Operation  Desert  Storm,  the  Army  quickly  upgraded  the  PAC-2  sys¬ 
tem  form  an  air  defense  system  with  the  Quick  Response  Program  (QRP)  and  Guidance 
Enhancement  Missile  (GEM)  modifications  in  1993  to  counter  the  new  SCUD  missile 
threat,  emerging  as  the  PAC-2  GEM  version  of  the  system.  The  current  PAC-3  system 
[MDA02v]  employs  HTK  technology  and  constitutes  the  lower  tier  of  a  layered  defensive 
capability  supporting  the  Theater  Missile  Defense  concept.  Major  PAC-3  components  in¬ 
clude  the  BMC4I,  sensors,  and  weapon  system.  In  2000,  the  system  was  preparing  for  In¬ 
dependent  Operational  Test  and  Evaluation  (IOT&E)  and  a  production  decision  [BRC+01]. 

2.  THEATER  AREA  AIR  DEFENSE  (THAAD)  PROGRAM  -  (1996- 
2001) 

THAAD  is  an  Army  theater  missile  defense  (TMD)  system  [MDA02u]  designed  to 
engage  the  full  spectrum  of  theater  class  ballistic  missile  threats,  employing  HTK  technol¬ 
ogy.  THAAD  was  designed  to  meet  critical  warfighters  needs  to  intercept  longer-range 
theater-class  ballistic  missiles  at  high  altitudes  and  further  away  from  the  intended  target. 
The  THAAD  system  is  the  only  defensive  system  that  is  effective  inside  and  outside  the 
earth’s  atmosphere,  and  may  be  quickly  deployed  worldwide  on  short  notice.  Major 
THAAD  components  include  the  BMC4I,  sensors,  and  weapon  system. 

3.  NAVY  AREA  DEFENSE  (NAD)  PROGRAM-  (1996-2001) 

The  NAD  is  a  lower  tier  system  designed  as  a  tactical  missile  defense  system 
against  endoatmospheric  missile  threats  within  the  earth’s  atmosphere.  The  Aegis  weapon 
system  equipped  with  the  phased  array,  SPY-1B/D  radar,  and  BMC4I  was  the  foundation 
weapon  platfonn  for  the  NAD.  It  will  evolve  into  the  future  sea-based  missile  defense  ca¬ 
pability  of  the  BMDS. 
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4. 


NAVY  THEATER- WIDE  (NTW)  DEFENSE  PROGRAM-  (1996-2001) 


The  NTW  missile  defense  system  [BMD99a]  was  a  tactical  missile  defense  system 
against  medium-and  intermediate-range  theater  ballistic  missiles  outside  the  earth’s  atmos¬ 
phere.  The  NTW  will  have  an  improved  interceptor,  phased-array  SPY-1B/D  radar,  and 
BMC4I.  With  the  evolution  of  the  modified  Standard  Missile  (SM-3)  and  enhancements  to 
the  Aegis  Weapon  System,  the  NTW  defense  system  has  become  a  component  of  the  sea- 
based  midcourse  segment  of  the  BMDS. 

5.  AIRBORNE  LASER  DEFENSE  PROGRAM  (ABL)-  (1996-2001) 

The  ABL  system  [MDA02o]  is  a  rapidly  deployable  airborne  platfonn:  a  747-400F 
aircraft,  equipped  with  a  long-range  laser  weapon  to  acquire,  track,  and  negate  TBMS  in 
the  boost  phase  of  flight.  The  ABL  will  be  capable  of  operating  above  cloud-level  for  ex¬ 
tended  periods  of  time,  providing  wide-area  geographic  and  near  24-hour  temporal  cover¬ 
age  of  potential  TBM  launch  areas.  The  ABL  weapon  system  includes  the  aircraft, 

BMC4I,  laser,  and  beam  control/fire  control  segments. 

6.  MEDIUM  EXTENDED  AIR  DEFENSE  SYSTEM  (MEADS)-  (1996- 

2001) 

The  Medium  Extended  Air  Defense  System  (MEADS)  [MDA02w]  is  an  interna¬ 
tional  cooperative  tenninal  missile  defense  program  shared  by  the  U.S.,  Germany,  and  Italy 
to  provide  360°  protection  for  maneuver  forces  against  short-range  tactical  ballistic  mis¬ 
siles,  cruise  missiles,  and  unmanned  aerial  vehicles.  MEADS  will  replace  HAWK  and 
PATRIOT  air  defense  systems  with  point  and  area  defense  capabilities.  The  system  is 
composed  of  a  surveillance  radar,  fire  control  radar,  launcher,  missile,  and  Tactical  Opera¬ 
tions  Center. 
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7. 


ARROW  WEAPON  SYSTEM-  (1996-2001) 


The  Arrow  Weapon  system  [MDA02t],  jointly  developed  by  the  U.S.  and  Israel, 
provides  short-and  medium-range  ballistic  missile  defense  of  Israel  and  U.S.  forces  de¬ 
ployed  in  the  region. 

C.  NATIONAL  MISSILE  DEFENSE  (NMD)  ORGANIZATION  -  (1996-2001) 

The  NMD  will  provide  protection  from  a  limited  attack  on  the  United  States  by  bal¬ 
listic  missiles.  The  NMD  program  [BMDOOc,  BMDOOi]  exploited  the  products  of  previous 
technology  and  development  programs,  including  the  ground-based  radar  (GBR)  prototype, 
development  of  the  X-band  phased-array  radar344,  passive  infrared  technology  from  the 
Brilliant  Pebbles  program,  and  advanced  tracking  algorithms  from  the  Exoatmospheric  Re¬ 
entry  Vehicle  Interceptor  System  (ERIS)  and  High  Endoatmospheric  Defense  Interceptor 
(HEDI)  programs.  The  NMD  system  [BMDOOc,  BMDOOi]  is  comprised  of  the  following 
major  components: 

1.  NMD  GROUND-BASED  SENSORS-  (1996-2001) 

Ground-based  sensors  include  the  Upgraded  Early  Warning  Radars  (UEWRS) 
[BMDOOh,  MDA02g],  based  on  existing  Pave  Paws  and  Ballistic  Missile  Early  Warning 
Systems  (BMEWS),  and  the  X-Band  Ground-Based  Radar  (GBR/XBR)345  system 
[BMDOOg], 

2.  NMD  SPACE-BASED  SENSORS-  (1996-2001) 

Space-based  Sensors  include  the  existing  Defense  Support  Program  (DSP)  satellite 
system346  [BMDOOh,  MDA02g],  the  planned  deployment  of  two  future  systems,  the  Space 
Based  Infrared  System  (SBIRS)  High  [BMDOOh,  MDA02g]  to  provide  missile  launch,  de- 
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XBR  efforts  included  improved  software  operatrions,  applications  processing,  and  new  radar  imaging  technologies  [BMDOOa], 

345 

The  XBR  phased-array  system  conducts  track-sweeping  operations  electronically,  which  is  must  faster  than  mechanically  based 
antennas.  With  high  resolution,  the  short  wavelength  of  the  X-band  radar  supports  early  tracking,  identification,  and  discrimination  of 
targets  some  2,400  mile  away  [BMDOOa] . 
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The  DSP  system,  initially  deployed  in  1971,  includes  a  constellation  of  satellites  in  geosynchronous  orbit  23,000  miles  above  the 
earth,  linked  to  earth  stations  to  provide  missile  launch  warning  and  limited  tracking  capabilities  [BMDOOa], 
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tection,  surveillance,  and  track  data;  and  the  Space  Based  Infrared  System  (SBIRS)  Low 
[BMDOOh,  MDA02g]  to  provide  more  discrete  identification,  discrimination,  and  tracking 
capabilities  early  in  the  missile’s  flight.  These  systems  will  also  provide  an  assessment  of 
post-intercept  results.  SBIRS  Low  builds  on  the  earlier  success  of  the  Midcourse  Space 
Experiment  (MSX),  which  demonstrated  the  capability  of  long- wave  infrared  surveillance 
and  discrimination  from  space  and  will  support  both  TMD  and  NMD  systems. 

3.  NMD  GROUND-BASED  INTERCEPTOR  (GBI)  -  (1996-2001) 

The  NMD  Ground-Based  Interceptor  is  an  enhanced  booster  stack  composed  of 
available  commercial-off-the-shelf  booster  components.  The  booster  delivers  the  Exoat- 
mospheric  Kill  Vehicle  (EKV),  a  kinetic  energy  HTK  interceptor  into  engagement  range 
for  a  midcourse  intercept  of  the  ballistic  missile  warhead  [BMDOOe,  MDA02q]. 

Battle  Management/Command,  Control,  and  Communications  (BM/C  )  system  is  actually  a 
composite  of  three  discrete  components:  battle  management,  command  and  control,  and 
communications.  Within  the  NMD  system  these  components  consist  of  the  software, 
hardware,  facilities,  and  communications  for  planning,  tasking,  and  controlling,  ensuring 
positive  human  control  over  all  aspects  of  the  NMD  system  [BMDOOa,  BMDOOf]. 
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A  ballistic  missile  is  an  intercontinental  class  missile  with  a  range  in  excess  of  5,500  kms  [BMDOOa], 

344 


APPENDIX  B  -  BALLISTIC  MISSILE  DELENSE  SYSTEM  (BMDS)  PROGRAM 

2002-PRESENT 

A.  BMD  SYSTEM  OBJECTIVES 

The  Agency  will  develop  the  Ballistic  Missile  Defense  System  incrementally  with 
layered  defenses  to  intercept  ballistic  missiles  of  all  ranges  in  all  phases  of  flight — boost, 
midcourse  and  terminal  [Rum02c]  as  illustrated  in  Figure  B-l. 


Missile  Defense  -  2002 
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ICBM/SLBM 
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Sea-Based  Kinetic 
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Space-Based  Directed  Energy 
Airborne  Directed  Energy 
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Passive  Defense 
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Figure  B-l.  The  BMDS  Mission  Scope  (After  [Gre03b]) 


The  existing  FoS  capabilities  will  continue  to  evolve  with  the  BMDS,  however  the 
combination  of  targets,  defense  systems,  and  layered  defensive  options  illustrated  in  Figure 
B-l  will  necessitate  major  changes  to  the  BMD  M&S  program.  The  BMDS  is  a  single  sys¬ 
tem  managed  as  an  integrated  program,  a  major  programmatic  shift  from  the  MDAP- 
centric  systems  previously  managed  by  the  Services.  The  BMDS  will  consist  of  elements 
configured  into  layered  defenses  to  provide  autonomous  and  mutual  support,  which  will 
provide  multiple  engagement  opportunities  against  a  threat  ballistic  missile  in  all  phases  of 
flight. 
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B. 


BMD  SYSTEM-LEVEL  CAPABILITY 


The  Department  provided  the  MDA  with  new  guidance  and  direction  identified  in 
Table  B-2  to  develop  the  BMDS. 


Previous  Direction 

New  Direction 

Treaty  Constrained 

Withdrew  From  ABM  Treaty  June  2002 

Requirements-Based 

Capability-Based 

Major  Defense  Acquisition  Program 
(MDAP)  Focus 

Ballistic  Missile  Defense  System  (BMDS)  Focus 

Joint  Requirements  Oversight  Council 

Senior  Executive  Council  (SEC)  And  Missile  Defense  Support 
Group  (MDSG) 

5000  Series  Acquisition  Life  Cycle 

3  Phases:  RDT&E,  Transition,  Procurement  By  Blocks 

Standard  DoD  Contract  Authority 

Other  Transaction  Authority  (OTA)  -  National  Team 

R&D  /  Acquisition  Functions 

R&D  /  Acquisition  Functions  /  IDO 

Mission  Needs  Statement  (MNS)  / 
Operational  /  Capstone  Requirements 
Documents  (ORD  /  CRD) 

System  Technical  Objectives  And  Goals  (TOG) 

Capability-Based  ORD 

Developmental  /  Operational  Test 

Developmental  /  Operational  Test 

Table  B-2.  Changes  to  BMDS  Direction 

The  BMDS  system  engineering  process  [MDA021]  will  allocate  these  system  capa¬ 
bility  specifications  cited  in  Table  B-3  to  BMDS  segments/elements/systems/  subsystems 
in  Table  B-4  through  a  series  of  documents  including  the  BMDS  Technical  Objectives  and 
Goals  (TOG),  the  Adversarial  Capability  Reference  Document  (ACRD),  System  Evolution 
Plan  (SEP),  Integrated  Master  Plan,  Integrated  Master  Schedule,  and  System  Capability 
Specification  (SCS),  among  others.  The  MDA  system  engineering  function  and  M&S  pro¬ 
gram  must  be  able  to  support  the  development,  maintenance,  and  testing  of  the  unprece¬ 
dented  BMDS  and  must  successfully  address: 

•  Operational  realities 

•  Engineering  complexities 

•  Transitioning  from  a  requirements-based  process  to  a  capabilities-based  approach 

•  Implementing  an  adversarial  capability  approach  instead  of  a  clearly  defined  threat 

•  Implementing  an  evolutionary  acquisition  strategy  using  a  two-year  block  approach 
|  MDA021 1. 
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MANAGEMENT 

COORDINATION 

ASSESSMENT 

•  Search 

•  Initial  ID 

•  Track 

•  Target 

•  Track 

•  Detect 

•  ID  Updates 

•  Situation 

Acquisition 

•  Discriminate 

•  Correlate 

•  Initial 

Awareness 

•  Weapons 

debris 

•  Track 

Classification 

•  LPE/IPP 

Assignment 

•  2d  Engagement 

•  Report 

•  Classification 

•  Target 

•  Weapons 

Opportunities 

•  Update 

Update 

Nomination 

Release 

•  Report 

•  Discriminate 

•  Discriminate 

•  Synchronize 

•  Mission  Control 

Status  /  Results 

•  Countermeasures 

•  Emerging  Threats 

•  Weapons 

•  Cue 

•  Cue 

•  Warhead  ID 

Management  •  Deconfliction 

•  Engagement 
at  Max  Range 

•  Upper/Lower  Tier 

Engagement 

•  Inventory  Management 

Table  B-3.  Required  BMDS  Capability 

C.  BMD  SYSTEM-LEVEL  LAYERED  DEFENSE  CAPABILITY 
1.  BOOST  PHASE  DEFENSE  SEGMENT  (BDS) 

The  Boost  Phase  Defense  Segment  (BDS)  [MDA02i,  MDA02j,  MDA02n]  is  de¬ 
signed  to  counter  that  part  of  a  threat  missile’s  flight  path  from  launch  until  it  stops  accel¬ 
erating  under  its  own  power.  At  this  point  in  its  flight  the  threat  missile  has  been  climbing 
against  the  Earth’s  gravity  for  100  to  300  sec.  and  may  achieve  an  altitude  of  200  km.  The 
Boost  Phase  Defense  (BDS)  intercept  of  the  missile’s  flight  by  the  ABL  [MDA02n]  under 
development  as  a  system  and  other  future  BDS  is  the  best  solution,  allowing  for  the  defense 
of  a  very  large  area  of  the  Earth,  and  destruction  of  the  target  before  the  deployment  of 
midcourse  countermeasures. 


2.  MIDCOURSE  DEFENSE  SEGMENT  (MDS) 

The  Midcourse  Defense  Segment  (MDS)  [MDA02i]  is  that  part  of  the  ballistic  mis¬ 
sile  trajectory  where  the  missile  has  completed  the  thrust  phase  and  is  following  a  more 

predictable  flight  path  during  the  mid-course  phase  as  it  freefalls  towards  its  target.  This 
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phase  may  last  as  long  as  20  minutes  in  the  case  of  an  intercontinental  ballistic  missile 
(ICBM).  This  phase  offers  the  defensive  system  a  longer  time  to  track  and  intercept,  and 
provides  the  possibility  of  having  enough  time  to  launch  more  than  one  interceptor.  How¬ 
ever,  this  phase  also  supports  the  deployment  of  countermeasures  against  the  defensive  sys¬ 
tem. 

The  Midcourse  Defense  Segment  has  two  systems:  the  Ground-Based  Midcourse 
Defense  (GMD)  [MDA02q]  and  the  Sea-Based  Midcourse  Defense  (SMD)  [MDA02r] 
elements,  successor  systems  to  the  NMD  and  NTW  programs.  The  GMD  interceptor  has 
two  major  parts:  the  booster  vehicle  and  the  Exoatmospheric  Kill  Vehicle  (EKV) 
[MDA02q].  The  EKV  destroys  the  target  with  kinetic  energy,  closing  on  the  threat  at  a 
closing  velocity  of  approximately  15,000  m.p.h.  at  altitudes  more  than  100  miles  above  the 
earth  [MDA02q].  The  SMD  includes  the  existing  Aegis  Weapon  Systems  (AWS)  with  the 
AN/SPY- 1B/D  radar,  and  the  Standard  Missile  3  [MDA02r]. 

3.  TERMINAL  DEFENSE  SEGMENT  (TDS) 

The  BMDS  Tenninal  Defense  Segment  (TDS)  [MDA02i,  MDA02t,  MDA03a]  cov¬ 
ers  the  final  phase  of  the  threat’s  flight  that  normally  lasts  approximately  30  sec.  for  an 
ICBM  class  of  vehicle  reentering  the  earth’s  atmosphere  at  over  2,000  M.P.H.  [MDA02j]. 
Countering  the  TDS  threats  are  the  PAC-3  missile  [MDA02v],  a  high  velocity  HTK  tech¬ 
nology  against  aircraft,  cruise  missiles,  and  short-range  ballistic  missiles,  the  Arrow 
Weapon  System  [MDA02t]  and  the  MEADS  [MDA02w].  The  THAAD  ground-based  sys¬ 
tem  complements  the  PAC-3  and  is  capable  of  engaging  short-and  medium-range  ballistic 
missiles  inside  and  outside  the  Earth’s  atmosphere  [MDA02u]. 

4.  SENSOR  SEGMENT 

The  Sensor  Segment  [MDA02i]  has  two  primary  projects:  Space  Sensors  and  Inter¬ 
national  Cooperation.  The  Space  Sensors  project  supports  the  Block  2010  Space  Based 
Infrared  System  (SBIRS)  increment  3  and  Space  Tracking  and  Surveillance  System  (STSS) 
and  will  also  support  the  BMDS  Boost,  Midcourse,  and  Terminal  Segments  with  a  constel¬ 
lation  of  infrared  sensing  satellites.  The  International  Cooperative  project,  the  Russian- 
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American  Observation  Satellite  (RAMOS)  [BMDOOa]  is  a  cooperative  research  and  devel¬ 
opment  effort  to  observe  the  earth’s  atmosphere  and  ballistic  missile  launches.  The  Targets 
and  Countermeasures  Program  [MDA02s]  provides  threat  representative  targets  and  coun¬ 
termeasure  challenges  to  the  evolving  BMD  system  and  elements. 


5.  BATTLE  MANAGEMENT  /  COMMAND  AND  CONTROL  (BM/C2) 


The  BM/C2  [MDA02i]  architecture  will  provide  a  highly  complex  and  advanced  C2 
element  to  effectively  manage  the  system  segments  and  execute  the  battle  management 
function  by  achieving  interoperability348  with  the  system  elements,  external  interfaces,  and 
integrate  with  the  national  military  command  structure.  BM/C2  improvements  include  al¬ 
gorithms  and  techniques  to  improve  the  control  of  the  future  missile  defense  system. 


WEAPONS 

SENSORS 

BMC2 

COMMS 

MISSILES 

RADARS 

BMC2 

COMMS 

PAC-2 

PAC-2  AN/MPQ-53 

BMDS  C2BM 

JDN/LINK-16 

PAC-2  GEM 

PAC-3  AN/MPQ-65 

GMD  BMC2 

LINK-16  JRE 

PAC-2  GEM  + 

THAAD  LOES/TPS-X 

AEGIS  BMC2 

JPN 

PAC-3 

THAAD  GBR 

PAC-2  BMC2 

JECN 

THAAD  PDRR 

COBRA  DANE 

PAC-3  BMC2 

JCTN 

THAAD  EMD 

UEWR 

ABL  BMC4I 

JMMN 

GMD  GBI 1  W/EKV 

SEA-BASED  XBR 

SBL  BMC2 

DISN 

GMD  GBI  II  W/EKV 

GBR/XBR 

JT  AGS/ALERT 

IBIS 

AEGIS  SM-2  BLOCK  IV 

AEGIS  SPY-1B 

MGS 

NAVY  LEAP  ALI 

ARROW  RADAR 

THAAD  BMC3 

SMD  SM-3 

MEADS  FIRE  CONTROL 

MEADS  BMC2 

ARROW 

MEADS 

MEADS  SURVEILLANCE 

SATELLITES 

DSP 

ARROW  BMC2 

TAOC 

CRC 

DIRECTED  ENERGY 

SBIRS-HIGH 

PLATFORMS 

AIRBORNE  LASER  (COIL) 

SBIRS-LOW 

YAL-1A  ABL  (747-400) 

SPACE-BASED  LASER 

RAMOS 

LASER 

THAAD  LAUNCHER 
PAC-2  LAUNCHER 
PAC-3  LAUNCHER 

KINETIC  ENERGY 

SEA-BASED  KINETIC 
SPACE-BASED  KINETIC 

BEACON  ILLUMINATOR 
TRACKING  ILLUMINATOR 
ACTIVE  LASER  RANGER 

AEGIS 

Table  B-4.  Major  Ballistic  Missile  Defense  Systems 
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Interoperability  is  the  ability  of  systems,  units,  or  forces  to  provide  services  to  or  to  accept  services  from  other  systems,  units,  or 
forces  and  to  use  the  services  so  exchanged  to  operate  effectively  together  [MDA02b], 
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D.  BMD  SYSTEM  ACQUISITION  STRATEGY 


The  new  Agency  mission  is  to  develop,  test,  and  prepare  for  deployment  of  a  mis¬ 
sile  defense  system  that  will  be  able  to  engage  all  classes  and  ranges  of  ballistic  missile 
threats  with  complementary  land-,  sea-,  air-,  and  space-based  interceptors,  directed  energy 
weapons,  sensors,  battle  management,  command  and  control,  and  communications  systems. 
The  BMDS  capability  identified  in  Figure  B-5  will  be  characterized  by: 

•  A  constantly  changing  system  configuration, 

•  Continuous  upgrades  to  software, 

•  A  constantly  changing  threat, 

•  Limitations  imposed  by  the  physical  environment  which  must  be  understood, 

•  Interoperability  and  integration  challenges,  and 

•  The  political  will  of  the  Nation. 

•  The  Agency’s  has  a  three-phased  evolutionary  acquisition  program  [MDA02k]: 
development,  transition,  and  procurement  and  operations  [Rum02c].  A  re¬ 
search,  development,  test,  and  evaluation  program  will  support  the  fielding  of 
missile  defense  capabilities  to  the  field  in  two-year  increments,  with  each  suc¬ 
cessive  block  having  a  greater  capability  than  the  previous  blocks. 


Figure  B-5.  BALLISTIC  MISSILE  DEFENSE  SYSTEM  (BMDS)  -  2002 
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E. 


BMD  SYSTEM  TECHNOLOGY  PLANNING 


Technology  played  a  critical  part  in  the  previous  missile  initiatives.  Future  required 
advances  in  technology  include  many  areas:  sensors349,  weapons350,  and  BMC3.  Critical 
sensor  area  research  includes  the  Advanced  Radar  Technology  program,  which  focuses  on 
anticipating  and  negating  countermeasures,  improving  discrimination  functions,  and  aids 
the  identification  of  the  target  in  a  potentially  cluttered  environment.  The  resolution  of 
many  complex  countermeasure  issues  requires  the  development  of  more  advanced  dis¬ 
crimination  algorithms  and  infrared  sensor  technologies. 

Future  weapon  technologies  include  research  into  Space-Based  Lasers,  Atmos¬ 
pheric  and  Exo-atmospheric  Interceptor  programs.  These  programs  will  incorporate  ad¬ 
vanced  HTK  technologies,  reduce  costs,  improve  seeker  capabilities  and  materials  of  the 
interceptor  to  discriminate  and  correctly  identify  the  correct  target  in  time  to  close  and  de¬ 
stroy  the  target. 


Sensors  and  detectors  working  individually  or  in  networks  may  include  passive  sensors  in  the  visible  (optical),  infrared,  and  ultravio¬ 
let  ranges;  active  sensors  including  radars  or  ladars  (laser  detection  and  ranging);  and  may  be  airborne,  ground-,  sea-,  or  space-based. 
Sensors  on  interceptors  and  satellites  are  normally  passive,  while  active  sensors  requiring  high  levels  of  energy  are  normally  surface- 
based  [BMDOOa], 
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Weapon  systems  may  be  airborne,  ground-,  sea-,  or  space-based,  and  must  deliver  enough  energy  into  the  target  to  destroy  or  negate 
it.  These  technologies  currently  include  HTK  interceptor  systems  and  directed  energy  (laser)  systems  [BMDOOa] 
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APPENDIX  C-  COMMANDERS  ANALYSIS  AND  PLANNING  SIMULATION 

(CAPS) 


The  CAPS  simulation  illustrated  in  Figure  C-l  consists  of  the  following  compo¬ 
nents  configured  in  separate  executable  files:  a  graphical  user  interface  (GUI),  a  simulation 
engine,  and  several  utility  programs,  which  convert  data  formats  and  support  analysis.  A 
user  may  enter  the  simulation  environment  or  run  the  simulation  through  the  GUI  or  via  a 
shell  script.  The  GUI  program  interfaces  with  the  other  CAPS  components  via  their  input 
and  output  files,  detects  program  termination  and  displays  the  results. 

CAPS  may  simulate  missile  defense  campaigns  up  to  several  months  in  length,  with 
a  scripted  striking  force,  and  a  defending  force  that  attempts  to  prevent  damage  to  defended 
areas  from  the  striking  force  ballistic  missile  threat  (Figure  C-2).  Developed  threat  scenar¬ 
ios  may  consist  of  a  series  of  raids  against  selected  targets  in  specified  regions  of  the  world. 
A  raid  may  consist  of  a  mix  of  ballistic  missile  and  air  breathing  threats  (e.g.,  aircraft, 
cruise  missiles),  termed  threat  objects,  available  to  the  striking  force.  Threats  may  be  vali¬ 
dated  threats  or  user-created/modified. 


CAPS  TODAY  IS  MDA’S 


Fast-Running,  High-Level,  Monte  Carlo 
Constructive  Simulation  Supporting 
Development  Of  The  Ballistic  Missile 
Defense  System  -  Employed  Primarily 
As  A  High-Level  Analysis  Of 
Alternative  And  Planning  Tool 


CAPS  Supports 
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Alternatives  And 
Trade  Space 


CAPS  Models  Ballistic  Missile 
Defense  System  Options  For 

•  Defense  Footprints 

•  Operating  Area 


•  Defended  Area 

•  Scenario  Performance 

CAPS  Supports  First- 
Order  Solution  And  Is 
HLA  Compliant 


Fast-Running,  Low  To 
Medium  Fidelity,  Constructive 
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Figure  C-l.  CAPS  Overview  (From  [Gre03a]) 
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System  libraries  for  many  current  systems  and  regional  topographic  data  are  pro¬ 
vided,  and  a  user  may  create  additional  systems  tailored  to  the  individual  study  require¬ 
ment.  Topographic  data  libraries  for  selected  regions  are  of  medium-high  resolution,  while 
the  entire  world  is  provided  at  lower  resolutions. 

Documentation  for  CAPS  includes  a  User’s  Manual  supporting  analysis  procedures, 
the  Methodology  Manual  [SpaOO]  describes  the  methods  and  algorithms  in  the  simulation 
engine,  the  Database  Guide  [SpaOl]  provides  the  data  structure  and  content  of  the  CAPS 
database,  and  the  CAPS  Training  Brief.  The  CAPS  system  also  provides  several  sources  of 
information,  including  a  statistical  summary  on  the  effectiveness  of  the  defensive  and  of¬ 
fensive  forces,  event  logs  from  previous  runs,  and  a  status  file  that  allows  follow-on  execu¬ 
tion  of  the  campaign  supported  by  CAPS  utility  programs  for  plotting  and  data  extraction. 


Defense  Operating  Defended  Scenario 

fmt|lmw  iVea A  m‘f<fi£itf9ifiience 


Figure  C-2.  CAPS  Intended  Uses  (From  [Gre03a]) 

The  defending  force  deploys  sensors  and  interceptors  in  space  and  the  air,  on  land, 
and  at  sea,  to  detect  and  engage  the  threat  objects  before  they  can  damage  defended  areas. 
Before  each  scenario,  the  defending  force  for  each  analysis  option  is  created  with  the  GUI, 
or  modified  from  a  previous  scenario.  The  CAPS  simulation  engine  reads  all  input  files, 
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initializes  the  campaign  system  clock  and  all  simulation  components,  before  entering  the 
main  simulation  loop. 

Within  the  main  simulation  loop,  CAPS  is  a  discrete-event  simulation  and  a  hybrid 
event-driven,  time-stepped  constructive  simulation  [SpaOO].  In  the  hybrid  event-driven, 
time-stepped  constructive  simulation  mode,  time-compression  algorithms  reduce  run  time 
in  scenarios  with  several  periods  of  intense  activity,  separated  by  long  periods  of  relative 
inactivity  prior  to  the  next  event.  (Figure  C-3). 

Using  time  compression,  CAPS  is  able  to  run  an  80-day  theater-level  campaign  in 
approximately  an  hour,  depending  on  the  available  machine  resources.  CAPS  was  de¬ 
scribed  by  [BMD99a]  as  a  low-fidelity  theater-level  analysis  tool,  capable  of  modeling  sys¬ 
tem  performance  and  interceptor  usage  for  land-,  air-,  and  sea-based  missile  defense  sys¬ 
tems  employed  against  TBMs. 

In  CAPS  the  primary  simulation  loop  contains  three  segments  (Status  Update,  Ac¬ 
quisition  and  Tracking,  Engagements)  and  acts  as  the  executive  [SpaOO].  Within  this  loop 
the  CAPS  phenomenological  models  that  simulate  the  campaign  environment,  threat  ob¬ 
ject,  sensors,  weapons,  and  command  authority  functions  simulate  the  events  that  occur 
within  the  current  step  [SpaOO]. 

The  first  segment  in  the  CAPS  Simulation  Engine  processes  the  status  of  all  defend¬ 
ing  units  and  all  of  the  threat  objects  through  a  call  to  the  CAPS  Composite  Threat  Model. 
This  process  activates  threat  objects  in  the  current  time  slot,  updates  a  state  vector  with  the 
threat  object  location  and  velocity,  and  deactivates  threat  objects  which  reach  their  target 
during  the  current  time  slot.  The  status  of  all  defensive  units  are  then  updated  by  calls  to: 

1)  the  CAPS  Deployment  Model  for  changes  to  sensor  or  interceptor  locations;  2)  the 
CAPS  Airborne  Interceptor  Model,  for  the  latest  status  of  airborne  interceptors;  and  3)  the 
CAPS  Interceptor  Model  that  provides  the  inventory  adjustment  status  for  interceptors 

The  last  task  of  this  segment  is  a  call  to  either  the  Phillips  Laboratory  Airborne 
Simulation  Tempest  Revision  (PLASTR)  model  or  the  ISAAC  model,  developed  by  the  Air 
Force,  that  simulate  the  suite  of  sensors  and  the  laser  weapon  of  the  ABL.  This  task  is  nec¬ 
essary  due  to  the  ABL’s  autonomous  search,  acquisition,  and  tracking  by  the  onboard  sen¬ 
sors  and  engagement  by  the  ABL  laser  weapon  [SpaOO]. 
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•  Graphical  User  Interface 

•  Input  And  Output  Databases  Are  Managed  Through  Tools  Provided 
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Figure  C-3.  CAPS  Architecture  (From  [Gre03a]) 

The  second  segment  determines  the  acquisition  and  tracking  of  threat  objects  by  de¬ 
fending  unit  sensors,  according  to  the  status  of  each  sensor  and  each  threat  object.  This 
segment  is  accomplished  by  calling  the  CAPS  Radar  Search  and  Acquisition  Model  to  pro¬ 
vide  the  search,  acquisition,  detection,  and  tracking  functions;  followed  by  a  call  to  the 
CAPS  Command,  Control,  and  Communications  Model  for  communication  link  status;  the 
CAPS  Composite  Threat  Model  to  update  state  vectors  for  active  threat  objects;  the  CAPS 
Radar  Signature  Model  to  detennine  whether  the  defensive  sensor  systems  actually  detect 
threat  objects  in  their  field  of  view;  and  lastly,  a  call  to  the  CAPS  Threat  Status  Model  to 
update  the  status  of  all  active  tracks  and  to  initiate  new  tracks  [SpaOO]. 

The  final  segment  processes  the  engagement  status  of  the  threat  objects  based  on 
the  results  of  the  previous  two  segments,  calling  the  CAPS  Battle  Management  Model  to 
initiate  new  engagements,  followed  by  a  call  to  the  CAPS  Interceptor  Engagement  Model 
to  update  engagements  in  process  or  when  it  creates  a  new  engagement.  Both  models  call 
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the  CAPS  Command,  Control,  and  Communications  Model  for  the  latest  status  about 
communication  links. 

The  CAPS  model  is  mostly  detenninistic,  with  the  exception  of  the  stochastic  radar 
detection  model  [BBG+99f].  CAPS  currently  runs  under  UNIX,  Windows,  and  Motif  on 
various  host  computers,  and  has  been  tested  on  Sun  SparcStations,  Silicon  Graphics  Indigo 
work  stations,  and  personal  computers  running  the  Linux,  Windows  NT  and  Windows 
98/2000  [SpaOO]. 
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APPENDIX  D  -MISSILE  DEFENSE  WARGAME  AND  ANALYSIS  RESOURCE 

(MDWAR) 


The  MDWAR  synthetic  battlespace  or  “gameboard”  supports  operators  with  a  real¬ 
time  constructive  simulation  of  futuristic  air  and  missile  defense  C2  systems  in  an  inte¬ 
grated  manner,  allowing  autonomous  and  interactive  missile  defense  C2  simulations  with 
variable  resolution  M&S  for  air,  ground,  space,  and  naval  forces  and  their  respective 
weapon  systems,  sensors  and  C2  [TRWOOb].  MDWAR,  illustrated  in  Figure  D-l  supports 
both  HLA  and  DIS  protocols  allowing  distributed  execution  and  interoperability  with  real 
C4I  systems  [TRWOOb].  MDWAR  provides  models  of  all  NMD  core  elements  and  spe¬ 
cific  TMD  models  for  all  of  the  four  JTAMD  pillars  of  the  active  defense. 


MDWAR  TODAY  IS  MDA’s. . . 


Virtual  Operator-In-The-Loop  For  Concept  Of 
Operations  (CONOPS),  Doctrine,  And  Tactics, 
Techniques  And  Procedures  (TTP)  Development  Using 
Operator-In-The-Loop  (OITL)  Perspective  For 
Warfighters 


MDWAR  Supports  Force-On- 
Force  Architecture 
Assessment  Analysis  And 
Combatant  Commander 
Assessments  And  Exercises 

HLA  /  DIS  Compliant 


MDWAR  Stimulates 
Discussion  Of 
Missile  Defense  Issues 


Real-Time,  Medium 
Fidelity,  Force-On-Force, 
Virtual,  OITL  Tool 


.  Simulating  Elements  Of  The 
Ballistic  Missile  Defense 
System 


.  In  A  Realistic,  Combat 
Environment 


Using  Human-In-Control 
Experiments 


.  Including  Real-World 
Confusion  From  Fog  Of  War 


.  Centrally  Managed  VV&A 
Program 


Figure  D-l.  MDWAR  Intended  Uses  (From  [Gre03a]) 

A  goal  of  the  MDWAR  C2  war  game  process  is  to  satisfy  approved  test  objectives, 
identify  the  human  usability  of  the  missile  defense  simulation,  and  use  this  feedback  to  de¬ 
velop  and  refine  the  CONOPS,  doctrine,  TTPs,  acquisition  strategy  and  military  utility  of 
the  evolving  BMDS.  The  operator  feedback  supports  the  evolution  of  weapon  designs,  in- 
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teroperability,  and  operational  procedures,  balancing  the  effectiveness  of  the  missile  de¬ 
fense  system  with  the  capabilities  of  human  operators  to  manage  the  system.  A  product  of 
the  MDWAR  experiment  process  is  the  engineering  data  to  support  system  designers,  op¬ 
erational  planners,  and  decision  makers  with  future  design  decisions,  CONOPS,  TTPs  and 
missile  defense  doctrine. 

MDWAR  simulates  a  large  number  of  objects  (~  1,000)  in  real-time,  with  sufficient 
detail  and  resolution  to  allow  for  discrete  differences  between  alternative  CONOPs  options 
for  operators,  analysts,  and  decision-makers,  and  supports  missile  defense  experiments  in 
five  different  modes: 

•  CONOPS  Development  -  This  is  the  primary  mode.  The  remaining  four  modes  are 
derivatives: 

•  Combatant  Commander  (formerly  CINC)  assessment  and  field  exercise, 

•  Staff  and  operator  familiarization, 

•  Missile  defense  architecture  assessment, 

•  Test  and  Evaluation  [TRWOOb]. 

MDWAR  replaced  the  Advanced  Real  Time  Gaming  Universal  Simulator  (AR¬ 
GUS)  model  due  to  requirements  for  higher  fidelity  weapon  and  sensor  models,  more  com¬ 
plex  and  realistic  games  with  player  positions  similar  to  those  in  the  field,  and  the  need  to 
reduce  the  turn-around  time  between  games  and  analyze  lessons  learned.  The  MDWARS 
architecture  (Figure  D-2)  is  a  distributed  architecture,  which  allocates  the  supporting  tasks 
to  computational  resources  separate  from  the  processors  required  to  support  the  mission- 
space  models.  MDWARS  is  data  driven,  versus  ‘hard-wiring’  or  internetworking,  employ¬ 
ing  parameter  files  for  missile,  radar  containing  quantity,  performance,  location,  and  other 
data,  upon  initialization  [TRWOOb]. 

In  order  to  enhance  the  performance  of  the  mission  space  models  and  the  proces¬ 
sors,  an  optimistic  parallel  discrete  simulation  (PDES)351  framework,  the  Agency  selected 
the  Synchronous  Parallel  Environment  for  Emulation  and  Discrete  Event  Simulation 
(SPEEDES)  as  the  simulation  executive.  To  achieve  performance  objectives,  SPEEDES 
executes  in  a  host  environment  featuring  processor  clustering,  network  computing,  and  par- 


PDES  simulations  are  categorized  by  how  time  is  managed,  how  time  management  functions  are  distributed,  if  distributed,  and  how 
the  distributed  components  interact  [TRWOOb], 
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allel  computing  architectures  employing  Symmetric  Multiprocessing  (SMP)352  and  Cache- 
Coherent  Non-Uniform  Access  (CC-NUMA)353  systems  [TRWOOb]. 
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Figure  D-2.  MDWAR’s  Architecture  (From  [Gre03a]) 


The  MDWAR  architecture  is  composed  of  three  sub-components  called  Top-Level 
Computer  Software  Components  (TLCSC),  which  are  executed  as  linked  applications: 

•  Missile  and  Air  Defense  Simulator  (MADSim)  TLCSC  -  This  TLCSC  is  composed 
of  the  mission-space  representations,  the  PDES  framework,  and  the  interfacing 
gateways  and  simulator  controls. 

•  Viewers  and  Editors  TLCSC  -  This  component  includes  simulation  control,  analy¬ 
sis  monitoring,  and  player  displays  supporting  the  human  interface  to  MADSim 
during  game  execution. 

•  MDWAR  Resource  Repository  TLCSC  -  This  component  maintains  all  the  data 
and  documentation  for  all  phases  of  the  game,  after  action  review  (AAR)  interfaces 
to  the  simulator’s  object-oriented  database,  and  analysis  support  tools  [TRWOOb] 
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“  On  a  SMP,  communication  between  processors  is  through  shared  memory  and  a  very  high  -speed  bus  [TRWOOb], 

353  In  a  CC-NUMA  architecture,  the  processors  access  shared  memory  through  a  cross-point  switch,  whereas  the  SMP  shares  both  mem¬ 
ory  and  data  buses  [TRWOOb] 
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APPENDIX  E  -  EXTENDED  AIR  DEFENSE  SIMULATION  (EADSIM) 


Since  1987  EADSIM,  shown  in  Figure  E-l,  has  evolved  and  today  provides  capabilities 
supporting  fixed  and  rotary-wing  aircraft,  ballistic  missiles,  cruise  missiles,  sensors,  command  and 
control,  communication  jammers354,  and  communication  networks.  EADSIM  supports  three  stan¬ 
dard  interface  protocols—  ALPS,  D1S,  and  HLA—  and  may  be  run  either  in  simulated  or  real  time, 
and  supports  message  passing,  event  passing,  and  control  passing,  via  these  standard  protocols 
[BAC+95].  EADSIM  is  compatible  with  Unix  operating  systems  on  Silicon  Graphics  workstations, 
Sun  workstations,  and  Windows  NT  [BBG+99e]. 


EADSIM  TODAY  IS  MDA’s  .  .  . 


System-Level  Ballistic  Missile 
Defense  Constructive  Simulation 
Modeling  Multiple  Functional  And 
Physical  Areas  Of  Interest 


J  Supports  War  Gaming,  BMD  Analysis,  And 
I  Exercises  For  Over  390  Worldwide  Users 


EADSIM  Models  Flight 
Movement, 

Communications,  Sensors, 
Jammers,  And  Weapons 


NH 


' 


Supports  Exercise 
Execution,  And 
Pre-  And  Post- 
Exercise  Analysis 

HLA,  DIS,  ALSP 
Compliant 


Medium  Fidelity,  Constructive  /  Virtual 
Simulation,  Flexible,  Composable, 
Tailorable,  Efficient  Simulation  / 
Stimulation 


...Stimulating  Tactical 
Hardware  And  Software 

...Of  The  Ballistic  Missile 
Defense  System  Elements  (-) 

...In  Realistic,  Simulated 
Environments 

...For  Interoperability 
Assessment  And  Integration 

...Emulating  The  Joint  Data 
Network  And  Comms 

...Centrally  Managed  Verification, 
Validation,  And  Accreditation 
Program 

Figure  E-l.  EADSIM’s  Intended  Use  (From  [Gre03a]) 

EADSIM  architecture  in  Figure  E-2  includes  the  following  general  models:  air  defense,  of¬ 
fensive  air  operations,  attack  operations,  multi-stage  ballistic  missiles,  air  breathers,  sensors,  jam¬ 
mers,  satellites,  early  warning,  generic  noncombatants,  communications,  electronic  warfare,  and 
specific  areas  of  interest  [BAC+95].  The  simulations  flexibility  is  a  key  reason  for  the  major  role 
EADSIM  plays  in  the  BMDS  cooperative  international  M&S  program.  EADSIM  performs  three 
basic  functions:  simulation  set-up,  scenario  execution  by  a  set  of  run-time  models  running  in  a 
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Jammers  are  radio  transmitters  accompanying  attacking  RVs  and  tuned  to  broadcast  as  the  defensive  radar  to  add  ‘noise’  to  signals 
reflected  from  the  RV  and  received  by  the  defensive  radar  [MDA02b], 
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multi-process  mode,  and  post-processing  and  analysis.  The  EADS1M  simulation  also  features  an 
integrated  graphic  user  interface  for  set-up  and  post-processing  support  [BAC+95].  EADS1M  ex¬ 
plicitly  models  each  platform.  Modeled  platforms  represent  a  system  type  (e.g.,  PAC-3)  and  may 
have  any  number  of  elements  such  as  sensors,  weapons,  defined  by  the  user  [JAS97c]. 


■  Integrated  Graphical  User  Interface 

■  Input  And  Output  Databases  Are  Managed  Through  Tools  Provided 

■  Interface  To  Other  Models  Via  DIS  /  HLA  /  ALSP 


Data  Input 


J 


Scenario 
^Displays 

Footprint 

^  f  - 

Sensor 


-  -Role  flayer  J 

rj  IntSff^.;+gfe_ 

jj.  -J.  T ifiith  1  perception  • 
^Display  ft  ^-+7 
•  Whiter  B4ue  /  Red 
Team  Ops 


Models 
•  EADTB 

•  JTLS 

•  MODSAF 

Live  Players 


Figure  E-2.  EADSIM’s  Architecture  (From  [Gre03a]) 

EADSIM  is  a  Monte  Carlo  model  with  a  capability  able  to  support  some  determi¬ 
nistic  analysis.  A  set  of  run-time  models  represent  selected  aspects  of  the  scenario,  support 
scenario  execution,  and  perfonn  data  exchanges  during  execution  [BAC+95].  There  are 
four  EADSIM  run-time  processes:  C3I,  flight  processing,  detection,  and  propagation 
[TOM97].  Major  functions  in  EADSIM  include  the  following:  battle  management,  com¬ 
munications,  movement,  propagation,  IR  and  radar  signatures,  radar  sensors,  and  natural 
environments.  All  models  in  EADSIM  rely  on  data  files  for  storage  and  retrieval  of  sce¬ 
nario  infonnation,  and  reflect  the  level  of  model  abstraction  maintained  in  the  data  files 
[Tom97]. 
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APPENDIX  F  -  EXTENDED  AIR  DEFENSE  TESTBED  (EADTB) 

EADTB  provides  a  capability  to  invoke  pre-compiled  algorithms  with  an  inter¬ 
preted  code,  which  provides  the  user  with  a  significant  capability  to  control  model  func¬ 
tionality  and  define  the  weapon  or  system  under  study  [BBB+96].  EADTB,  a  two-sided 
model  (Red  and  Blue  forces),  illustrated  in  Figure  F-l,  models  many  types  of  air  defense 
weapon  systems,  with  algorithms  supporting  the  varying  degree  of  representation  detail 
required;  and  features  the  ability  to  model  perception  versus  truth  data,  that  allows  the  in¬ 
corporation  of  limited,  flawed,  or  misinterpreted  information  to  influence  the  decision 
process  [BBB+96].  Missile  defense  functional  areas  modeled  by  EADB  include:  BM/C2, 
movement,  missile  launching,  lethality,  vulnerability,  signature  generation,  communica¬ 
tions,  and  the  natural  environment  [Ray02a].  With  its  variable  levels  of  detail,  it  is  able  to 
support  studies  of  systems,  components,  tactical  doctrine  development,  and  mission  plan¬ 
ning  [BMD99a]. 


EADTB  TODAY  IS  MDA’s  .  .  . 
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Figure  F-l.  EADTB’s  Intended  Uses  From  [Gre03a]) 
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SSRs,  called  rule  sets,  are  a  major  feature  of  EADTB’s  capability  and  flexibility,  al¬ 
lowing  EADTB  to  function  at  various  levels  of  detail  and  aggregation  [BBB+96,  MR01], 
The  EADTB  Common  Model  Set  (CMS)  maintains  SSR  algorithms,  and  provides  back¬ 
ground  processes  and  routines  called  by  SSRs  [BBB+96].  User-developed  SSRs,  built 
from  EADTB  components  (e.g.  Thinker,  Platfonn,  Communications,  and  Sensor)  represent 
specific  weapon  systems  supported  by  the  modeled  natural  environment  in  which  these 
components  operate. 

Each  SSR  has  at  least  one  or  more  Thinker  components  [Wai94,  WS94],  which 
provides  the  intelligence  for  the  SSR,  the  constructive  surrogate  for  the  human  command 
and  control.  Thinker  rule  sets  act  on  perception355  data  as  opposed  to  truth356  data.  Thinker 
algorithms  are  available  to  rule  sets  through  a  set  of  specific  symbols.  The  Thinker  re¬ 
ceives  data  from  onboard  sensors  and  by  way  of  the  communications  network.  However 
EADSIM  processes  “truth”  data  by  sending  communication,  sensors,  and  other  elements 
information  transmitted  to  the  Thinker  as  “perceived”  data.  Integral  to  the  Thinker  are  the 
user-defined  Rule  sets  [ZCE+01]. 

The  rule  set  logic  combines  with  the  representation  of  the  command  and  control 
system  and  available  communication  networks  to  simulate  the  C2  process.  The  Rule  set 
invokes  a  series  of  algorithms  to  complete  functions  including  threat  assessment,  missile 
guidance,  correlation,  fusion,  and  weapon  assignment.  EADTB  supports  a  wide  range  of 
potential  SSRs  supported  by  user-defined  rule  set  logic,  which  may  represent  both  manual 
and  automatic  processes.  The  EADTB  interpreted  rule  set  in  EADTB  consisting  of  a  set  of 
symbols  and  syntax  rules  provide  all  C2  decision-making  logic,  and  supports  simulation 
scenario  changes  without  recompiling  or  linking.  The  flexibility  inherent  in  EADTB  and 
the  rule  sets  also  creates  the  greatest  challenges,  since  it  requires  so  much  input  from  the 
users  of  the  tool. 

A  SSR  also  has  one  Platform  component.  The  Platform  component  contains  algo¬ 
rithms  to  allow  it  to  model  movement  and  other  physical  characteristics  of  different  Plat- 

355 

‘Perception’  data  constitutes  the  results  of  perception  transformations  (observations,  derivation,  or  estimates)  by  given  entities  of  the 
states  of  other  entities  within  the  scope  of  the  simulations  /  exercise  as  manifest  in  local  perceived-information  data  structures  (possibly 
corrupted,  inaccurate,  imprecise,  or  imperfectly  known)  [Wai94] 

356  ‘Truth’  data  is  generated  as  part  of  the  original,  intrinsic  unequivocal  representation  within  the  scope  of  the  simulations  /  exercise  and 
which  constitutes  the  truth  regarding  the  states  of  the  entities  as  known  to  themselves  or  the  executive  [Wai94], 
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form  types,  different  kill  mechanisms,  Platfonn  signatures,  and  Platfonn  vulnerabilities. 
The  Platfonn  component  has  five  constituent  capabilities  or  physical  characteristics  that 
correlate  to  the  real  world:  1)  move  /  motion  capabilities,  2)  the  ability  to  cause  damage,  or 
3)  be  damaged,  4)  the  ability  to  carry  and/or  launch  an  object,  and  5)  the  ability  to  generate 
a  signature  for  reflection  or  emission  [Ray02a]. 

A  Communication  component  allows  the  transfer  of  data  between  Thinkers,  but 
does  not  access  the  contents  of  the  message.  Determining  who  may  communicate  is  dy¬ 
namic  since  the  operational  state  of  communication  devices  may  change.  However, 
EADTB  delivers  only  error-free  messages  from  the  receiving  communication  unit  to  the 
receiving  Thinker.  Two  communication  models  support  message  transfers,  a  low-detail 
model  called  Direct  Comms  for  statistically  modeling  message  delay,  and  the  Standard 
Comins  model  which  models  the  links,  queues,  networks,  and  propagation  factors 
[Ray02a].  The  two  methods  for  dissemination  messages  are  point-to-point  and  broadcast. 

Sensor  components  model  the  detection  of  objects  by  measuring  energy  levels  re¬ 
flected  or  emitted  by  the  object.  Target  detection  is  a  function  of  the  sensor  perfonnance, 
target-signature  characteristics,  and  environmental  effects.  The  user  defines  the  sensor  per¬ 
fonnance  by  selecting  the  sensor  power  and  gain.  Sensor  models  may  be  active  sensors 
such  as  radar358  or  passive  sensors  including  EO/iR  passive359  and  RF  passive  sensors360 
[Ray02a].  A  sensor,  controlled  by  rule  set  commands,  and  tasked  by  a  Thinker,  may 
transmit  and/or  receive  over  a  common  set  of  user-defined  extended  frequencies. 

The  level  of  sensor  detail  ranges  from  functional  sensors  that  only  process  range, 
target  type,  and  search  volume;  to  fine-grained  sensor  models  that  can  model  gain, 
polarization,  Doppler  spreads,  ambiguous  ranges,  and  jamming  (main  lobe  or  side  lobe) 
[Ray02a].  The  Sensor  component  is  comprised  of  a  set  of  models  used  to  simulate  the  fol¬ 
lowing  sensors:  radar,  RF  passive,  RF  noise  jammer,  EO/iR  passive,  and  functional  sensor; 
which  are  further  decomposed  into  four  subcomponents:  scanner,  transmitter,  receiver,  and 
data  processor  [Ray02a]. 


357  The  signature  may  be  in  the  radio  frequency  (RF),  electro  optical  (EO)  or  infrared  (IR)  wavelengths. 

358  Radars  transmit  RF  energy  and  measure  its  reflection  from  the  body  of  the  target  [Ray02a], 

359  Sensors  that  measure  the  energy  emitted  by  the  body  of  the  target  [Ray02a], 

360  Sensors  that  measure  the  energy  emitted  from  an  active  sensor  transmitter  carried  on  the  target  [Ray02a], 
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The  Environment  algorithms  model  the  portion  of  the  environment  that  affects  ob¬ 
jects  by  three  feature  groups:  1)  terrain  and  sea  surface,  2)  atmosphere,  and  3)  celestial  bod¬ 
ies.  The  simulation  supports  DTED,  DFAD  and  user-derived  data  sets  [Ray02a].  The  en¬ 
vironment  influences  sensor  and  communication  representations,  sensor  detection,  and  the 
performance  of  object  movement  representations. 


Specific  System  Representation  (SSR)  a 


EADTB  Entities  Are  Built  From  Components,  Allowing  The  User 
The  Flexibility  To  Create  Representations  Of  Systems  At  The  Detail 
Of  The  Component  Pieces 


Time  Slot  Allocation 


Rulesets  Are  Isolated  From  One  Another  So 
That  All  Communications  Are  Performed 
Through  The  Communications  Model. 
Rulesets  Cannot  Access  Simulation  Ground 
Truth,  Thus  Ensuring  The  Perception-Based 
Paradigm 


EADTB  Is  DIS  And  HLA  Compliant  -  Additionally  Provides  The  Ruleset 
Interfaces  To  External  Processes  For  BM/C2  SWIL 


Figure  F-2.  EADTB’s  Architecture  (From  [Gre03a]) 

EADTB  analysts  build  scenarios  incrementally,  developing  elements  such  as  air¬ 
craft,  communication  devices,  and  sensors.  When  deployed  on  a  digitized  terrain  map,  the 
systems  become  platfonns,  and  then  combined  into  lay  downs,  and  lastly,  merged  into  sce¬ 
narios.  EADTB  architecture  shown  in  Figure  E-2,  runs  on  a  quad-processor  Covex  3840, 
supporting  Silicon  Graphics  workstations  and  successfully  ported  to  Silicon  Graphics  and 
Origin  2000  workstations.  EADTB  is  currently  DIS  and  HLA  compliant  [SAP98,  Wai99, 
STB+00,  WS00]. 


368 


APPENDIX  G  -  MISSILE  DEFENSE  SYSTEM  EXERCISER  (MDSE) 

Starting  with  a  proof  of  principle  in  1993,  and  evolving  with  an  incremental  build 
methodology,  MDSE,  shown  in  Figure  G-l,  exercises  real  tactical  hardware  and  software 
in  dynamic,  interactive  interoperability  tests  allowing  the  hardware  to  be  operated  earlier  in 
the  development  cycle  and  over  a  wider  range  of  conditions  than  may  ever  be  possible  in 
live  tests.  A  simulated  secure  voice  radio  environment  between  tactical  operators  at  each 
site  support  MDSE  FIWIL  tests,  while  test  operators  have  secure  voice  communications 
coordination  with  recording  and  playback  capabilities.  MDSE  uses  the  DIS  protocol 
[McQ97], 


MDSE  TODAY  IS  MDA’s  .  .  . 


:cji 

■  t< 


Ballistic  Missile  Defense 
System-Level  Distributed 
Real-Time  Hardware-In-The- 
Loop  (HWIL)  Tool 


MDSE  Supports  Event  HWIL  Planning, 
I  Control,  And  Assessment 


MDSE  Monitors  HWIL 
Truth,  Situational 
Awareness,  And 
Activity  Data 


MDSE  Supports 
HWIL  Post-Test 
Analysis 


<1 


DIS  Compliant 


Real-Time,  Virtual 
Simulations,  Centrally  Controlled, 
Geographically  Distributed, 

( .  I’S  Synchronized  HWIL  To of 


.  Stimulating  Tactical 
Hardware  And  Software 


.  Of  The  Ballistic  Missile 
Defense  System  Elements  (-) 

.  In  Realistic,  Simulated 
Environments 


.  For  Interoperability 
Assessment  And  Integration 


.  Emulating  The  Joint  Data 
Network  And  Comms 

.  Centrally  Managed  (SES) 
Verification,  Validation, 
Accreditation  Program 


Figure  G-l.  MDSE’s  Intended  Uses  (From  [Gre03a]) 

As  a  virtual  model,  MDSE’s  objective  is  to  stimulate  the  actual  missile  defense 
hardware  and  software  to  the  maximum  extent  possible  and  employ  embedded  constructive 
simulations  only  when  necessary  due  to  cost,  availability,  or  physics.  Areas  where  con¬ 
structive  simulations  in  MDSE’s  architecture  (Figure  G-2)  are  employed  include:  the  infra¬ 
red  and  radar  sensor  front  ends;  transmitter,  receiver  and  signal  processing;  interceptor  fly 
out;  and  portions  of  the  BMC3  [McQ97].  MDSE  also  emulates  the  Joint  Data  Network 
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(JDN)  [YSOO]  with  a  gateway  in  order  to  pass  Tactical  Digital  Information  Link-J  (TADIL- 
J)  [DoD97,  BSB+02],  Tactical  Information  Broadcast  System  (TIBS),  Tactical  and  Related 
Applications  (TRAP),  and  Tactical  Data  Distribution  (TDDS)  tactical  messages  to  exercise 
participants  [Too94]. 

MDSE  then  stimulates  the  sensor  data  processing  systems  of  the  tactical  software 
and  hardware  to  develop  a  flow  of  infonnation  through  the  tactical  system  interfaces.  The 
stimulation  supports  an  assessment  of  FoS  interoperability  issues  [BB98a,  BB98b],  includ¬ 
ing  reporting  responsibility  (R2),  track  management,  cueing  effectiveness,  engagement  co¬ 
ordination,  and  tactical  battle  management  and  C2  implementation  in  the  participating  sys¬ 
tems  [Too94].  The  stimuli  for  the  track  processing,  based  on  realistic  representations  of 
pre-scripted  theater-level  attack  scenarios,  pass  over  a  geographically  distributed  network 
to  a  remote-environment  interface  connected  to  the  tactical  system  [WasOl]. 

MDSE  is  composed  of  three  segments  interconnected  by  the  Test  Communications 
Network  (TCN),  the  MDSE  Control  Segment  (TCS),  the  Tactical  System  Driver  Segments 
(TSDs),  and  the  Tactical  Communications  Environment  Segment  (TCES).  The  TCN  con¬ 
sists  of  a  combination  of  MDSE  component  and  segment  local  area  networks  and  wide  area 
networks  interconnected  by  T 1  encrypted  lines  in  a  star  topology  between  the  TEC  at  the 
hub  to  all  the  remote  environments  (RE)  at  the  test  sites  [McQ97].  The  TCN  allows  MDSE 
to  monitor  and  affect  the  health  and  welfare  status  of  the  MDSE  network. 
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■  Gateway  Interfaces  /  T-l  Circuit 

■  Intra-Segment  interfaces 

-  Test  Control  Interfaces  /  T-l  Circuit 
Blue  Text  Constructive  Model 


Build  3.4  To  4.0/4. 1 
Upgrade 


Figure  G-2.  MDSE’s  Architecture  (From  [Gre03a]) 


The  Tactical  Communications  Environment  Segment  (TCES)  emulates  the  Joint 
Data  network  (JDN)  and  enables  the  tactical  systems  to  exchange  tactical  messages  consist¬ 
ing  of  emulated  TADIL-J  communication  link  messages  in  a  simulated  communication  en¬ 
vironment,  that  result  from  the  injection  of  truth  data  into  the  various  systems  [TEM00]. 
The  TCES  consists  of  the  Link- 16  Gateway  System  for  JDN  emulation  and  emulation  of 
the  TIBS  and  TDDS  messages.  The  Link- 16  Gateways  also  provide  the  TEC  with  message 
display,  monitoring,  and  recording  capabilities  with  a  Central  Node  Gateway,  at  the  HWIL 
test  control  site,  and  a  Remote  Node  Gateway  at  each  one  of  the  participating  test  sites. 

The  MDSE  TCS  segment  simulates  the  environment  of  the  operational  architecture 
and  provides  overall  control  of  the  HWIL  test.  The  TCS  consists  of  a  Test  Exerciser  Con¬ 
trol  (TEC)  at  the  HWIL  test  control  location,  and  an  RE  at  each  of  the  participating  test 
sites.  A  Theater  Environment  (TE)  node,  a  real-time  simulation  of  the  physical  environ¬ 
ment  within  which  the  tactical  systems  will  operate,  is  located  within  the  TEC  and  at  each 
RE.  Physical  phenomena  include  threat  object  propagation  and  debris,  system  location, 
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propagation  of  natural  objects  such  as  weather,  terrain,  natural  and  hostile  environments 
and  auxiliary  interfaces  such  as  the  global  positioning  system  (GPS). 

A  major  function  of  the  TE  node  is  to  provide  tactical  systems  only  that  information 
which  the  systems  would  naturally  perceive  through  their  sensors  or  which  affect  their  op¬ 
erational  perfonnance.  By  segregating  simulation  truth  data  from  tactical  system  percep¬ 
tion,  MDSE  ensures  that  that  a  tactical  systems  performance  is  not  biased  by  information 
that  would  not  be  available  to  the  tactical  system  under  the  simulated  scenario,  since  the 
tactical  system  must  react  to  the  TE  node  stimulus  as  it  would  to  a  real-world  situation 
[McQ97], 

The  MDSE  TSD  segments  accept  truth  data  from  the  TCS  Theater  Environment 
(TE),  detennine  whether  the  sensors  can  see  the  object(s)  as  scripted  in  the  scenario,  and  if 
appropriate,  convert  the  truth  data  into  signals  compatible  with  the  systems  sensor.  The 
TSD  segment  consists  of  the  tactical  driver,  which  interfaces  between  the  REs  and  the  tac¬ 
tical  system  or  simulator.  The  MDA  program  office  responsible  for  the  tactical  system  de¬ 
velopment  provides  the  system  driver  through  which  MDSE  interfaces,  stimulates,  and 
controls  the  test  execution  for  the  tactical  system. 

The  system  drivers  provide  the  physical  interface  to  the  system  under  test,  receiving 
data  from  the  MDSE  TCS  segment,  and  convert  the  data  to  a  fonnat  usable  by  the  tactical 
system.  System  drivers  may  also  provide  simulations  of  tactical  components  of  the  weapon 
system  or  sensor,  required  by  the  scenario,  but  may  be  missing  or  unavailable  for  a  HWIL 
test  [McQ97].  The  hardware  for  the  development  and  operational  environments  includes 
Sun  Ultra’s,  Sun  Servers,  Silicon  Graphics  Indigos,  and  X-terminals. 
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APPENDIX  H  -  ELEMENT  LEVEL  M&S 


Table  H-l  is  a  listing  of  the  major  Element-level  M&S  systems  to  provide  a  context 
of  the  constructive  and  HWIL  Elements  M&S  systems  within  the  MDA  M&S  Hierarchy. 


BMDS  ELEMENT  LEVEL  CONSTRUCTIVE  M&S 

PATRIOT 

THAAD 

GMS 

SMS 

ABL 

KEB 

PAC2SIM 

ETESIM 

LIDS 

JHU-6  DOF 

ISAAC 

TRITON  E2E 

PAC3SIM 

TISES 

SENTRY 

KWEVAL 

LAMDA 

KWEVAL 

DDSE 

MEDUSA 

SSTB 

HYDROCODES 

SIMPROCESS 

MFSIM 

THHADSIM 

TESS 

MEDUSA 

BMAP 

FMS-D 

STAT 

TEX 

SSTRESS 

ICONOS 

TEAM-2 

RELEX 

ACCSIS 

ABLPROP 

SEEKER 

SIMTAS 

FIRMTRACK 

HITS 

ASTAB 

XBR  DIGSIM 

ARTEMIS-T 

UEWR  DIGSIM 

MARS 

UTB 

NMDSIM 

WPNSIM 

BMDS  ELEMENT  LEVEL  HARDWARE-IN-THE-LOOP  (HWIL) 

FMS 

SIL 

ISTCl 

GSEL 

TACCSF 

GTSF 

TEC 

ISTC2 

RAY  HIL/CIL 

MSS-3 

TRTB 

PCIL 

CSEDS 

LMMFC 

TTC 

ETEDDS 

TMTD 

[PAT01] 

[THA02] 

[TCD+00] 

[AEG02] 

[ABL01] 

[KEB02] 

Table  H- 1 .  MAJOR  BMDS  ELEMENT  LEVEL  M&S 
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APPENDIX  I  -  THREAT,  ENVIRONMENT,  SIGNATURE,  and  LETHALITY 

(TSEL)  MODELS 

Appendix  I  provides  brief  descriptions  of  the  BMDS  System-Level  Threat,  Signa¬ 
ture,  Environment,  and  Lethality  M&S.  These  models  support  the  development  and  testing 
of  the  Element-Level  M&S  listed  in  Appendix  H.  and  the  BMDS  System-Level  Core 
M&S. 

A.  BMDS  SYSTEM-LEVEL  THREAT  M&S 

1.  TM93 

TM93  is  a  modeling  and  simulation  architecture  used  by  the  SPC  to  provide  intelli¬ 
gence  credible  representations  of  threat  missile,  aircraft,  and  cruise  missile  systems.  The 
TM93  Threat  Model  is  the  official  set  of  tools  used  within  the  Special  Program  Center 
(SPC)  to  generate  threat  files  of  ballistic  threats  and  the  responsibility  for  assuring  that  the 
threat  scenarios  used  in  simulations,  studies,  and  analyses  adequately  represent  the  best  as¬ 
sessment  of  the  intelligence  community. 

The  SPC  has  the  responsibility  for  the  integration  and  execution  of  the  various  tools 
used  to  generate  standard  threat  tapes.  The  TM93  Threat  Model  is  the  set  of  integrated 
tools  used  to  produce  standard  threat  tapes  and  includes  tools  to  allocate  strategic  and  thea¬ 
ter  ballistic  threats  and  air-breathing  threats.  The  allocation  tools  feed  tools,  which  gener¬ 
ate  high  fidelity  trajectories  of  all  threat  objects  including  boosters,  tanks,  PBV's,  RV's,  de¬ 
bris,  etc.  Post-processing  tools  perform  quality  control  checks  and  to  feed  the  final  docu¬ 
mentation  process.  All  threats  include  a  complete  set  of  hardcopy  documentation  of  the 
scenario.  TM93  provides  representations  of  single  events  through  mass  strategic  attacks. 
TM93  models  threat  trajectories  using  an  ECI  coordinate  frame,  J4  oblate  earth  model,  and 
standard  atmosphere. 

The  TM93  documentation  includes  a  User’s  Manual.  TM93  uses  a  UNIX  based 
operating  system.  However,  certain  applications  only  operate  on  IRIX/Silicon  Graphics 
platforms.  A  conversion  tool  exists  allowing  data  conversion  to  DIS  PDUs.  TM93  will 
meet  HLA  standards  in  the  future. 
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2.  STRATEGIC  AND  THEATER  ATTACK  MODELING  PROCESS 

(STAMP) 

STAMP  is  a  workstation-based  threat  generator  and  engineering  analysis  tool  that 
models  ballistic  missile  flights  from  launch  to  impact.  STAMP  is  a  missile  flight  generator 
and  engineering  analysis  tool  that  models  missile  flight  from  launch  to  impact  and  presents 
extensive  flight  characteristics  and  trajectory  descriptions  using  a  wide  array  of  graphical 
and  tabular  outputs.  It  features:  an  easy  to  use  operator  interface  using  Windows. 

STAMP  is  missile-database  driven,  developed  and  approved  by  intelligence  agen¬ 
cies  and  contains  the  parameters  and  values  needed  to  model  strategic,  theater,  and  space- 
launch  missiles  consistent  with  intelligence  estimates.  These  databases  are  in  a  standard 
fonnat,  read  directly  into  STAMP.  The  main  output  from  STAMP  consists  of  state  vectors 
in  a  specified  fonnat. 

STAMP  has  three  primary  applications:  (1)  missile  analysis;  (2)  scenario  genera¬ 
tion;  and  space-launch  vehicle  modeling.  The  modeling  and  analysis  tool  assists  with  mis¬ 
sile  definition  development,  checkout,  and  depiction  of  missile  characteristics  and  flight 
trajectories  to  include  penaids  and  sub  munitions.  The  scenario  modeling  and  analysis  tool 
generates  simple  or  massive  attack  scenarios  and  provides  graphical  and  statistical  charac¬ 
teristics  of  the  scenarios.  The  space-launch  vehicle-modeling  tool  allows  users  to  specify 
the  desired  satellite  orbit.  STAMP  will  determine  the  missile  flight  path  to  achieve  this  or¬ 
bit. 

STAMP  has  a  wide  range  of  missile  modeling  capabilities  that  satisfy  the  needs  of 
most  users  for  accurate  and  timely  missile  performance  information  in  functional  fonnats. 
The  plotting  and  data  outputs  from  STAMP  have  evolved  over  several  years  of  threat 
analysis  support  and  have  been  continually  refined  to  meet  user  needs.  The  primary  appli¬ 
cations  are:  (1)  Missile  analysis,  (2)  Scenario  generation,  and  (3)  Space-launch  vehicle 
modeling. 

Documentation  includes  the  STAMP  User’s  Manual,  STAMP  Design  Document, 
Missile  Definition  Data  Base  (MDDB)  Files,  Worldwide  Ballistic  Missile  Baseline  Per¬ 
formance  Characteristics,  ACE  User’s  Manual,  and  NTF  Interface  Requirements  Specifica- 
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tion.  STAMP  source  code  includes  Ada  and  C,  and  it  runs  on  UNIX  platforms  (e.g.,  Sun 
and  Silicon  Graphics),  with  two  support  graphics  packages  (XMGR  and  AXIS). 

The  following  are  inputs  to  STAMP:  Missile  Definition  Data  Bases  (MDDB);  facility  files; 
scenario  specifications;  modeling  parameters;  earth  model;  Digital  Integrated  Combat 
Evaluator  (DICE)  system  files;  Modular  Vehicle  Simulation  (MVS)  aeronautical  data.  In 
addition  to  the  graphic  and  tabular  outputs,  STAMP  can  also  generate  every  object  threat 
files  and  MEM  threat  input  files. 

STAMP  links  to  the  following  models:  TISES  (THAAD  System  Simulation);  PTOS 
(PATRIOT  Tactical  Operations  Simulation);  BESIM  (Brilliant  Eyes  Simulation);  PERM 
(Penaid  Effectiveness  and  Requirements  Model);  MEM  (Mission  Effectiveness  Model);  6 
DOF  free  flight  models  (ATAP  &  MVS);  Commercial  Satellite  Analysis  codes  (STK  and 
OMNI);  chemical  and  biological  codes.  The  Space  Warfare  Center  also  uses  STAMP  to 
model  all  ballistic  and  space-launched  missiles  for  input  to  their  sensor. 
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B.  BMDS  SYSTEM-LEVEL  SIGNATURE  AND  ENVIRONMENT  M&S 

1.  SYNTHETIC  SCENE  GENERATION  MODEL  (SSGM) 

The  SSGM  generates  composite  Infrared  (IR)  scenes,  X-band  radar  cross  sections 
and  IR  LIDAR  cross  sections  and  range/Doppler  returns  using  high  fidelity,  validated  stan¬ 
dard  component  phenomenology  models.  The  SSGM  consists  of  computer  software  and 
input  databases  that  provide  the  capability  to  generate  two-dimensional  time-sequenced 
scenes  as  well  as  other  supporting  data  for  use  in  the  design,  development,  and  testing  of 
surveillance  and  weapons  systems. 

The  SSGM  computes  viewer-perspective  apparent-radiance  pixel  maps  and  point- 
source  intensity  data  from  the  output  of  state-of-the-science  phenomenology  codes  and  au¬ 
thenticated  input  databases.  This  SSGM  performs  via  an  interactive  software  system  that 
assists  users  in  creating  the  scene-sequence  scenario  and  content,  selects  or  generates  the 
required  databases,  and  renders  the  databases  to  generate  the  desired  output.  In  addition, 
provided  tools  allows  users  to  display  and  analyze  the  output  images.  The  standard  wave¬ 
length  regime  extends  from  the  UV  to  the  far  infrared,  plus  the  extension  to  include  hard- 
body-modeling  capability  at  radar  frequencies.  The  space-time  samplings  are  those  of  the 
sensor  systems  and  are  not  otherwise  constrained. 

The  SSGM  developers  restructured  the  software  in  Release  6.0  to  make  it  easier  to 
add  and  remove  capabilities  from  the  overall  system.  The  SSGM  uses  the  concept  of  a 
“component”  to  describe  the  different  phenomenology  model  capabilities.  The  SSGM  has 
five  types  of  components:  general,  foreground,  background,  laser  foreground,  and  non¬ 
imaging.  General  components  include  the  Observer,  Environment  and  Trajectory;  which 
provide  infonnation  to  the  models  as  required.  The  various  phenomenology  models  in¬ 
clude  foreground,  background,  and  laser  foreground  or  non-imaging  components. 

SSGM  radiance  output  scenes  consist  of  quiescent  and  enhanced  natural  and  per¬ 
turbed  backgrounds  with  embedded  foreground  elements  such  as  targets  and  target  related 
events.  Background  components  include  terrain,  horizon,  earth  limb,  aurora,  space,  and 
nuclear  events.  Foregrounds  include  boosting  missile  plumes  and  hardbodies,  and  post¬ 
boost  hardbody  objects  including  RVs  and  decoys.  Clouds  maybe  modeled  as  either  back- 
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ground  or  foreground  components.  The  hardbody  LIDAR  model  is  the  laser  foreground 
component.  All  radar  components  are  non-imaging. 

SSGM  contains  numerous  first-principles  physics  models.  As  such,  it  contains  a 
number  of  assumptions  and  approximations.  The  principal  limitation  is  the  spectral  cover¬ 
age  and  spatial  resolution.  Most  SSGM  imaging  components  cover  the  0.2-25 pm  wave¬ 
length  range.  Plumes  and  nuclear  phenomena  are  restricted  to  a  2-25pm  range.  Most  im¬ 
aging  components  support  a  spatial  resolution  of  >10-9  radians.  Celestial  backgrounds  are 
limited  to  >4  x  10-5  radian  resolution.  Terrain  and  cloud  databases  have  inherent  resolu¬ 
tions  that  vary  from  30  -  18500  m  and  30  -  1000  m,  respectively.  These  effectively  bound 
sensor  footprint  resolution.  Primary  applications  include: 

•  Sensor  Development 

•  Sensitivity  &  Trade-Off  Analyses 

•  Threat  Signature  Development 

•  Nominal  Threat  Signature,  Launch  through  impact 

•  Realistic  range  and  distribution  of  signatures 

•  Algorithm  Development 

•  Fusion 

•  Detection,  discrimination  and  tracking 

•  Data  Analysis 

•  Test  and  Evaluation 

•  Test  planning  and  excursion  evaluation 

•  Hardware-in-the-loop  simulations  and  emulators 

•  High-level  simulations 

•  Support  for  Wargames:  signatures  and  backgrounds 

•  System  Architecture  Studies 

The  SSGM  main  architecture  shown  in  Table  1-1  includes  OOA/OOD  techniques 
and  C++.  Phenomenology  component  codes  include  a  mix  of  FORTRAN,  C  and  C++. 
Interface  between  the  component  codes  and  SSGM  involves  C++  “wrappers”.  The  GUI 
written  in  C++,  uses  X-Windows  and  X-Motif  call.  Component  model  source  code  is  gen¬ 
erally  not  available  for  modifications,  as  any  modification  might  negate  the  validation 
status  of  the  model.  The  Hardware  includes  Silicon  Graphics  workstations  or  servers  run¬ 
ning  IRIX. 
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Scenarios  can  take  seconds  to  days  to  execute  to  run  to  completion.  This  is  princi¬ 
pally  a  function  of  the  number  and  complexity  of  the  targets,  types  of  background  (e.g., 
clouds,  earthlimb),  phenomenology  models  selected  and  number  of  scenes  to  be  generated. 

Current  Inputs  include  Scenario  Description  File  (all  other  necessary  databases  are 
internal).  This  includes  sensor,  target  and  engagement  description,  but  does  not  include 
specific  target  trajectories.  Current  Outputs  include 

•  At-aperture  radiance  scenes 

•  Target  and  plume  radiant  intensities 

•  Target  and  sensor  metrics 

•  Target  heating  and  ablation  history 

•  Detailed  information  on  atmosphere,  clouds,  and  terrain 


Atmosphere 

Nuclear 

SHARC 

IRSim 

MODTRAN 

HiSEMM 

SAMM 

PEM 

SAG 

Celestial 

APART 

MOSART 

IRAS  Star  Catalog 

FASTLIMB 

CBSKY4 

CIRRIS-1A 

CBZODY 

CBAMP 

Clouds 

TROs 

CLDSIM 

osc 

Aurora 

OPTISIG 

AURORA 

Xpatch 

DELTAS 

Terrain 

Missile  Plume 

GENESSIS 

SIRRM 

Vents/Spills 

Debris 

CHARM 

Fuel  Vent 

KIDD-OPTISIG 

CAMERA 

Vent-Trail 

DEBRA-KIDD 

SPF-2 

Table  1-1 .  SSGM  Constituent  Model  Matrix 
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2.  THE  BATTLESPACE  ENVIRONMENT  SIGNATURE  TOOLKIT 

(BEST) 

The  Battlespace  Environment  Signature  Toolkit  (BEST)  program  will  be  developed 
to  replace  the  legacy  Environment  and  Signature  models  shown  in  Table  1-1  and  Table  1-2, 
establishing  the  foundation  for  the  next-generation  missile  defense  Scene  Generation  capa¬ 
bility  for  future  advanced  sensor  development  based  on  requirements  identified  by 
[CKB+94,  Hec99,  Hie99,  SB99,  BerOOa,  BMC+00,  BSB+00,  BurOO,  BVW+00,  CWM+00, 
DKC+00,  DNC+00,  DSHOO,  DWE+00,  FAK+00,  KMM+00,  MSB+00,  PBOO,  SST+00]. 
The  MDA  awarded  a  contract  in  June  2002  for  the  development  of  the  Battlespace  Envi¬ 
ronment  and  Signatures  Toolkit  (BEST)  development  program  supporting  the  design,  de¬ 
velopment,  testing  and  evaluation  of  active  and  passive  sensors,  signature  processing,  and 
algorithms  with  empirical  and  physics-based  signature  phenomenology  and  battle  space 
environment  codes  for  all  threes  phases  of  missile  flight. 

Meeting  the  requirement  to  counter  the  full  range  of  future  potential  threats,  the 
MDA  developed  an  Adversarial  Capability  methodology  which  will  be  introduced  into  the 
MDA  System  Core  Models,  augmenting  the  existing  DIA  validated  Design-to-Threat,  pro¬ 
viding  BMDS  system  engineers  with  new  capabilities  for  trade  space  analysis. 


BMDS  Legacy  Environment  and  Signature  Codes 

Model/Code 

Function 

Executing  Agent 

Target  Related  Hardbody  Objects 

osc 

Resolved  Passive  EO/IR  Signatures 

SMDC 

OPTISIG 

Unresolved  Passive  EO/IR  Signatures 

SMDC 

XPATCH 

Radar  Signatures  (Physical  Optics) 

AFRL/SN 

FISC 

Radar  Signatures  (Method  of  Moments) 

AFRL/SN 

RTS 

Radar  Signatures 

NRL 

DELTAS 

Laser  Retroreflectance 

SMDC 

Missile  Plumes  &  Trails 

SFM,  VIPER,  TDK 

Flowfield,  Nozzle  &  Combustion 

AFRL/PR 

SPF,  GIFS 

Complex  Plume  Flowfield 

AFRL/PR 

PERCORP.SPP 

Liquid  &  Solid  Motor  Performance 

AFRL/PR 

SIRRM 

Plume  Radiance  (<  50  km) 

AFRL/PR 

CHARM 

Plume  Flowfield  &  Radiance  (>  50  km) 

AFRL/PR 

SPURC 

Plume  UV/VIS  Radiance  (>  50  km) 

AFRL/PR 

SOCRATES 

Monte  Carlo  Flowfield  &  Gas  Dynamics 

AFRL/PR 

HALT,  PARCS 

Laser  backscatter  &  Plume  RCS 

AFRL/PR 
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Battlespace  Environment 

MODTRAN 

Atmospheric  Transmission  &  Radiance  (<50  km) 

AFRL/VS 

SHARC 

Quiescent  Atmosphere,  Auroral  &  Terminator  Radi¬ 
ance  and  Clutter  (>50  km) 

AFRL/VS 

SAMM 

Atmospheric  Transmission  &  Radiance  (all  altitude) 

AFRL/VS 

SAG,  MSIS,  BEAM,  SIG 

Atmospheric  State  and  Clutter 

AFRL/VS,  NRL  &  MSIC 

CBSD 

Planets,  Moons,  Asteroids,  Zodiacal  Emission,  Stars 

AFRL/VS 

GENESSIS 

Terrain  &  Ocean  Surface  Radiance 

AFRL/VS 

CLDSIM 

Cloud  Radiance,  Transmission,  &  Glints 

AFRL/VS 

IRSIM,  HiSEMM,  NucSim, 

NORSE 

Nuclear  Perturbed  Atmosphere 

DTRA 

Trajectory 

STAMP 

Strategic  Missile  Trajectories 

DIA/NAIC 

DICE 

Theater  Missile  Trajectories 

DIA/MSIC 

Integrated  Models 

SSGM 

Targets,  Trajectories,  and  Environments 

NRL 

PLEXUS 

Expert  System  for  Atmospheric  &  Celestial  Phe¬ 
nomenology 

AFRL/VS 

CHAMP 

Missile  Plume  &  Hardbody 

AFRL/MN 

Table  1-2.  BMDS  Signature  and  Environment  Functions 


Because  of  the  distributed  nature  of  BMD  weapon  systems  and  the  critical  impor¬ 
tance  of  each  element  to  achieve  overall  system  effectiveness,  interoperability  and  collec¬ 
tive  system  perfonnance  assessments  are  an  important  area  of  evaluation.  All  BMD  sensor 
and  weapon  systems  require  signature  information  upon  which  to  base  decisions.  Existing 
MDA  signature  codes  have  served  the  engineering  analysis  community  over  the  years,  but 
they  are  difficult  and  expensive  to  modify,  difficult  for  the  user  to  learn  and  operate  cor¬ 
rectly,  and  most  importantly  cannot  effectively  support  the  future  M&S  needs  of  MDA. 
Present  and  anticipated  requirements  will  demand  greater  interoperability,  consistency,  and 
faster  execution  speeds.. BEST  will  enhance  signature  tools  and  complete  the  phenomenol¬ 
ogy  set  effectively  providing  a  robust  signature  toolkit  for  all  MDA  signature  needs.  It 
will: 

•  Provide  a  more  complete  representation  of  battlespace  signatures. 

•  Ensure  consistency  in  signature  results  thereby  assisting  verification  and  validation 
and  accreditation. 

•  Meet  the  future  challenge  of  simulation-driven  design,  evaluation,  and  acquisition. 

•  Provide  users  the  flexibility  they  require 

•  Support  interoperability  (within  the  signature  codes  and  with  external  simulations). 
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BEST  will  replace  the  patchwork  assemblage  of  signature  codes  with  a  consistent, 
modular  code  that  is  accessible  through  one  interface.  BEST  will  support  the  entire  range 
of  missile  defense  signature  needs.  The  modular  nature  of  BEST  will  support  plug-and- 
play  of  functionally  equivalent  components.  This  will  pennit  contractors  to  insert  proprie¬ 
tary  components  and  to  even  use  the  components  delivered  with  BEST  in  other  applica¬ 
tions.  The  following  illustration  depicts  the  component-based  structure  of  BEST. 

A  secondary  goal  of  BEST  is  to  provide  the  government  with  signature  tools  as  part 
of  a  suite  of  government-approved  tools  to  independently  evaluate  and  assess  contractor 
perfonnance. 
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C.  BMDS  SYSTEM-LEVEL  LETHALITY  M&S 

The  BMD  System  lethality  model  program  consists  of  PEELS  and  PEGEM  as  a 
suite  of  tools  continually  evolving  to  identify  any  residual  threat  after  a  successful  inter¬ 
cept.  The  BMD  System  lethality  programs  directly  support  the  development  of  the  MDA 
element  level  M&S  and  system  level  testing. 

1.  POST  ENGAGEMENT  GROUND  EFFECTS  MODEL  (PEGEM) 

The  Post-Engagement  Ground  Effects  Model  (PEGEM)  is  a  simulation  tool  to  study 
ground  effects  caused  by  chemical,  biological,  and  conventional  weapons/agents  distrib¬ 
uted  in  both  unitary  (bulk)  and  submunition  (canister  or  bomblet)  fonn.  The  model  pro¬ 
vides  chemical/biological  agent  ground  contamination  and  conventional  weapons  blast 
zones.  It  also  provides  data  for  unit  effectiveness  and  casualty  estimation. 

PEGEM  is  a  suite  of  models  to  produce  a  simulation  incorporating  various  phases 
of  ground  effects  assessment,  such  as  Munition  Propagation  and  Atmospheric  Transport. 

In  a  typical  case,  the  analyst  specifies  a  chemical  or  biological  weapon  event  scenario  in¬ 
cluding  all  threat  details  and  the  locations  and  times  of  the  various  events  on  the  user- 
defined  grid.  Intercept  lethality  information  flows  from  the  output  of  an  endgame  lethality 
model.  The  lethality  model  provides  PEGEM  a  prediction  of  the  fraction  of  payload  sur¬ 
viving  following  an  intercept  event.  For  canister  submunition  payloads,  the  locations  of 
surviving  submunitions  within  the  target  payload  are  given.  PEGEM  uses  this  information 
to  propagate  the  potential  residual  threat(s)  to  the  ground. 

PEGEM  documentation  includes  a  User’s  Manual,  Technical  Manual,  and  other 
supporting  documentation.  Written  in  ANSI-C,  PEGEM  runs  on  Windows  95,  Windows 
NT  and  UNIX 
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2.  PARAMETRIC  ENDO-  EXO-ATMOSPHERIC  LETHALITY 

SIMULATION  (PEELS) 

PEELS  is  a  high  fidelity  terminal  endgame  lethality  simulation,  which  models  hit- 
to-kill  and  fragment  kill  of  ballistic  missile  from  kinetic  energy  weapon  attack.  It  operates 
as  a  non-real-time  simulation,  intended  to  support  weapon  system  designers  as  well  as  sys¬ 
tem  level  analysts.  The  models  primary  purpose  is  to  analyze  lethal  effectiveness  of  non¬ 
nuclear  kill  interceptors  in  an  anti-tactical  ballistic  missile  role. 

PEELS  is  a  tenninal  endgame  lethality  simulation,  which  models  hit-to-kill  (HTK) 
and  fragment  kill  of  ballistic  missiles  from  Kinetic  Energy  Weapons  (KEW)  attack. 

PEELS  predicts  damage,  probability  of  kill  (Pk),  and  submunitions  kill  fraction  dependent 
on  target  design  using  lethality  criteria  developed  for  a  number  of  threats.  The  design  fa¬ 
cilitates  force-on-force  systems  studies  where  six-degree-of-freedom  (6-DOF)  fly  out  mod¬ 
els  predict  endgame  state  vectors. 

Algorithms  in  the  PEELS  code  also  allow  parametric  investigations  of  lethality 
through  variation  of  endgame  variables  such  as  miss  distance,  crossing  angle,  warhead 
burst  point,  intercept  altitude,  target  and  interceptor  velocity  and  angles  of  attack,  target 
aim  point,  and  fragment  pattern  density.  The  model  evaluates  HTK  and  fragment  kill  ef¬ 
fects  against  both  high  explosive  (HE)  and  chemical  unitary  warheads,  and  against  both  HE 
and  chemical  submunitions  warheads.  Primary  applications  areas  include:  Parametric  Fea¬ 
sibility  Analysis  Studies — warhead  concept  evaluation;  System  Assessment — scenario  spe¬ 
cific  lethality  coupled  with  a  6-DOF  model;  Pre-Test  Analysis — help  define  test  matrix — 
identify  stressing  cases  to  be  tested. 

Documentation  includes:  An  Executive  Summary  and  an  eight-volume  set  of 
documentation.  Volume  0,  Executive  Summary;  Volume  I,  PEELS  Technical  Manual  (Al¬ 
gorithm  and  model  detail);  Volume  2,  PEELS  Targets  Manual  and  Volume  3,  TMD  Lethal¬ 
ity  Criteria  (Infonnation  describing  the  geometric,  structural,  and  payload  design  of  the  tar¬ 
gets  and  lethality  criteria);  Volume  4,  PEELS  Validation  and  Verification  Manual;  Volume 
5,  User’s  Manual;  Volume  6,  Accreditation  Report;  Volume  7;  Maintenance  and  Distribu¬ 
tion  Manual;  Volume  8,  Source  Code  Manual. 
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PEELS  runs  on  SGI  UNIX  based  system,  VAX,  or  PC  systems,  and  programmed  in 
FORTRAN-77.  PEELS  operates  in  two  basic  modes  of  operation:  1)  the  Database  Mode 
and  2)  the  Direct  Mode.  The  Database  Mode  uses  a  pre-processor  to  create  databases  ac¬ 
cessed  during  execution,  for  rapid  system  level  applications  where  lethality  is  only  one  part 
of  many  computations.  The  Direct  Mode  runs  if  the  target  model  does  not  exhibit  axial- 
symmetry  or  the  desired  fragment  type  is  not  in  the  database.  This  typically  requires  four 
to  five  times  more  computer  processing  time. 

Database  Mode  requires  a  series  of  databases  accessed  during  execution.  These 
contain  information  about  the  solid  geometry  of  the  target  as  well  as  parameters  associated 
with  fragment  lethality.  An  Index  Database  composed  of  shot-line  intersection  with  inter¬ 
nal  target  components  maintains  data  considered  critical  to  target  lethality.  The  index  da¬ 
tabase  points  to  a  Lethality  Database.  The  lethality  database  maintains  empirical  data. 

This  contains  infonnation  that  defines  the  minimum  fragment  material  and  energy  charac¬ 
teristics  necessary  to  achieve  lethality  along  each  shot  line. 
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APPENDIX  J  -  THE  DOMAIN  METADATA  REPOSITORY  REGISTRY  (DMRR) 


A.  COMMON  DMRR  METADATA  FACILITIES 

From  [IS079c]361  we  extract  the  metadata  facilities.  This  is  a  critical  component 
for  achieving  Registry  interoperability  with  the  domain,  inter-domain  integration,  and  at  the 
enterprise  level.  The  common  DMRR  facilities  illustrated  in  Figure  J-l  support  adminis¬ 
tered  items,  identified  once  within  the  registry.  The  named  administered  items  have  at  least 
one  context,  maybe  more,  and  within  each  context,  names  and  definitions  specified  in  one 
or  more  languages.  The  administered  items  have  classification  in  zero  or  more  classifica¬ 
tion  schemes  [IS079c]. 


Figure  J-l .  BMDS  DMRR  Common  Facilities  (From  [IS079c]) 


©  International  Organization  for  Standardization  (ISO).  This  material  is  reproduced  from  1S0/IEC  1 1179-3,  Second  Edition,  2003- 
02-15  with  permission  of  the  American  National  Standards  Institute  on  behalf  of  ISO.  No  part  of  this  material  may  be  copied  or  repro¬ 
duced  in  any  form,  electronic  retrieval  system  or  otherwise  or  made  available  on  the  Internet,  a  public  network  by  satellite  or  otherwise 
without  the  prior  written  consent  of  the  American  National  Standards  Institute,  25  West  43rd  Street,  New  York,  NY,  10036. 
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B. 


DMRR  TYPES  OF  ADMINISTERED 


[ISO079c]  specifies  and  describes  the  types  of  Administered  Items362  listed  in  Fig¬ 
ure  J-2,  and  allows  definition  of  additional  types  of  Administered  Items. 


Figure  J-2.  BMDS  DMRR  Types  of  Administered  Items  (From  [IS079c]) 


36~  An  Administered  Item  may  be  one  of  the  types  listed  in  Figure  J-2.  Each  instance  of  the  Administered  Item  encapsulates  its  own 
Administration  Record.  An  Administered  Item  is  submitted  by  an  Organization  represented  by  the  relationship  submission  in  Figure  J-4. 
A  Registration  Authority  registers  Administered  Item  represented  by  the  relationship  registration  in  Figure  J-4.  An  Organization  admin¬ 
isters  the  Administered  Item  represented  by  the  relationship  stewardship  in  Figure  J-4.  A  Reference  Document  may  describe  zero  or 
more  Administered  Item  represented  by  the  relationship  reference  in  Figure  J-4.  Each  instance  of  the  Administered  Item  through  an 
Administered  Record  has  a  unique  Administered  Item  Identifier  [lS097c], 
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c. 


DMRR  HIGH-LEVEL  METAMODEL 


Figure  J-2  provides  a  high-level  overview  of  the  central  regions  of  the  metamodel. 
[IS079c] 
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Figure  J-3.  BMDS  DMRR  High-Level  Metamodel  Overview  (From  [IS079c]) 


D.  DMRR  ADMINISTRATION  AND  IDENTIFICATION  METAMODEL  RE¬ 
GION 

The  Administration  and  Identification  in  Figure  J-4  and  Figure  J-5  supports  the  ad¬ 
ministrative  areas  of  the  Administered  items  in  the  DMRR  including: 

•  Identification  and  registration  of  items  submitted  to  the  registry, 

•  Organizational  information  and  responsible  agencies,  including  Registration  Au¬ 
thorities363, 

•  Organizational  contact  information, 

•  Supporting  documentation, 

•  Relationships  among  administered  items  [IS079c] 


363  A  Registration  Authority  is  any  organization  authorized  to  register  metadata.  A  Registration  Authority  is  a  subtype  of  organization 
and  inherits  all  of  its  attributes  and  relationships.  An  Administered  Item  has  a  Registration  Authority  that  is  its  owner,  shown  by  the 
relationship  in  Figure  J-4.  A  Registration  Authority  may  register  many  Administered  Items  [lS097c]. 
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Figure  J-4.  BMDS  DMRR  Admin  /  Identification  Metamodel  Region  (From  [IS079c]) 

E.  DMRR  CLASSES  USED  AS  COMPOSITE  DATATYPES 

[IS097f]  prescribes  he  registration  of  Administered  Items.  Figure  J-5  shows  the 
composite  data  types  used  on  composite  attributes. 


Reqistration  Authority  Identifier 

Administration  Record 

international_oode_designator  [1  .1]  :  String 
organization_identifier  [1  ..1]  String 
organ ization_part_identifier  [0.  1]  :  String 

OPI_source  [0  .1]  String 

administered_item_identifier  [1 .1  ]  :  ltem_ldentifier 
registration_status  [1 .1]  :  String 
administrative_status  [1.1]  :  String 
creation_date  [1..1]  Date 
last_change_date  [0..1]  :  Date 
effecti ve_date  [0.1]  :  Date 

La  nguage_lden  tific  ation 

language_identifier  [1 .  1  ]  String 
country_identifier  [0  .1]  String 

until_date  [0  1]  :  Date 
change_description  [0..1]  :  String 
administrative_note  [0..1]  :  String 
explanatory_comment  [0.  1  ]  .  String 

Contact 

contact_name  [11]  :  String 
contact_title  [0..1]  String 

unresolved_issue  [0.  1]  String 
origin  [0..1]  :  String 

contact_information  [1  .1]  :  String 

ltem_ldentifie  r 

item_registration_authority_identifier  [1..1]  Registration_Authority_ldentifier 
data_identifier  [1.1]:  String 
version  [1 .1]  :  String 

Figure  J-5.  BMDS  DMRR  Classes  /  Composite  Datatypes  in  the  Administration  and 

Identification  Region  (From  [IS079c]) 
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DMRR  NAMING  AND  DEFINITION  METAMODEL 


Figure  J-6  represents  the  Naming  and  Definition  Region  used  to  manage  the  names 
and  definitions  of  administered  items  and  the  contexts  for  the  names.  An  administered  item 
may  have  many  names  that  will  vary  based  on  discipline,  locality,  and  technology  em¬ 
ployed  [IS079c].  [IS079d]364  provides  rules  and  guidelines  for  the  fonnulation  of  data 
definitions,  and  [IS079e]  provides  naming  and  identification  principles  for  administered 
items  within  a  context. 


Figure  J-6.  BMDS  DMRR  Naming  and  Metamodel  Region  (From  [IS079c]) 


364 

©  International  Organization  for  Standardization  (ISO).  This  material  is  reproduced  from  1S0/IEC  1 1179-4,  First  Edition,  1996-07- 
01  andlSO/IEC  1 1179-5  First  Edition,  1995-12-01  with  permission  of  the  American  National  Standards  Institute  on  behalf  of  ISO.  No 
part  of  this  material  may  be  copied  or  reproduced  in  any  form,  electronic  retrieval  system  or  otherwise  or  made  available  on  the  Internet, 
a  public  network  by  satellite  or  otherwise  without  the  prior  written  consent  of  the  American  National  Standards  Institute,  25  West  43rd 
Street,  New  York,  NY,  10036. 
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G. 


DMRR  CLASSIFICATION  METAMODEL  REGION 


The  Classification  Region  illustrated  in  Figure  J-7  provides  a  facility  to  register  and 
administer  Classification  Schemes365  and  their  constituent  Classification  Scheme  Items.  In 
addition,  a  Classification  Scheme  may  classify  Administered  Items  within  the  registry. 
Some  Classification  Schemes  will  be  more  applicable  to  classifying  objects  in  the  real 
world  than  classifying  metadata  objects  in  the  registry  [IS079c]. 


Figure  J-7.  BMDS  DMRR  Classification  Metamodel  Region  (From  [IS079c]) 


365  A  classification  scheme  may  be  a  taxonomy,  a  network,  an  ontology,  or  any  other  terminological  system;  or  the  classification  scheme 
may  consist  of  a  list  of  controlled  vocabulary  words  and  terms.  A  classification  scheme  is  a  sub-type  of  Administered  Item,  inherting  its 
attributes  and  relationships,  supporting  identification,  naming,  definition,  and  classification  functions  [lS079c]. 
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H.  DMRR  DATA  ELEMENT  CONCEPT  METAMODEL  REGION 


The  data  element  concept  region  illustrated  in  Figure  J-8  maintains  the  information 
on  the  concepts  upon  which  the  data  elements  are  developed.  The  metadata  objects  in  this 
region  concentrate  on  semantic  concepts.  The  metadata  objects  in  this  region  are  object 
classes  and  properties,  combined  to  form  Data  Element  Concepts.  The  concepts  are  inde¬ 
pendent  of  any  internal  or  external  physical  representation.  [IS079c].  A  Data  Element 
Concept  may  be  represented  in  the  form  of  a  data  element,  described  independently  of  a 
particular  representation,  with  zero  or  one  object  class  and  zero  or  one  property.  A  prop¬ 
erty366  is  a  characteristic  common  to  all  members  of  an  object  class. 


Figure  J-8.  BMDS  DMRR  Conceptual  Metamodel  Region  (From  [IS079c]) 


366  A  property  may  be  any  feature  that  humans  naturally  use  to  distinguish  one  individual  object  from  another.  It  is  the  human  percep¬ 
tion  of  a  single  characteristic  of  an  object  class  in  the  real  world.  It  is  conceptual  without  an  associated  means  of  representation  for 
communication  [IS079c]. 
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I.  DMRR  CONCEPTUAL  AND  VALUE  DOMAIN  METAMODEL  REGION 


The  region  of  the  metamodel  in  Figure  J-9  supports  the  administration  of  the  Con¬ 
ceptual  Domains  and  Value  Domains,  viewed  as  logical  code  sets  and  physical  code  sets. 
Conceptual  Domains  support  Data  Element 367  Concepts368 ,  and  Value  Domains  support 
Data  Elements  as  illustrated  in  Figure  J-9.  A  Conceptual  Domain  is  a  set  of  Value  Mean¬ 
ings,  which  may  be  expressed  or  enumerated  by  a  description,  and  as  an  Administered  Item 
maintains  Administrative  Record  information,  allowing  identification,  naming,  definition, 
and  optional  classification  within  a  Classification  Schema  [IS079c]. 

A  Conceptual  Domain  may  be  associated  with  other  Conceptual  Domains  through  a 
Conceptual  Domain  Relationship,  either  as  a  composition  with  other  Conceptual  Domains, 
or  as  a  component  member  of  a  larger  Conceptual  Domain.  A  Conceptual  Domain  may 
specify  a  constraint  such  as  a  linear  measure  as  its  dimensionality.  A  specified  dimension¬ 
ality  requires  that  any  Value  Domain  based  on  the  Conceptual  Domain  specify  a  Unit  of 
Measure369 .  An  Enumerated  Conceptual  Domain,  a  sub-type  of  Conceptual  Domain  may 
contain  a  finite  number  of  enumerated  notions  (e.g.,  codes  representing  the  names  of 
states).  Conversely,  a  Non-enumerated  Conceptual  Domain  lacks  a  finite  set  of  Value 
Meanings.  As  a  sub-type  of  the  Conceptual  Domain,  both  the  Enumerated  Conceptual 
Domain  and  Non-enumerated  Conceptual  Domain  inherit  the  attributes  and  relationships  of 
the  former  [IS079c]. 

In  an  Enumerated  Conceptual  Domain,  each  member  has  a  Value  Meaning,  in  the 
registry  distinguishing  it  from  other  members,  and  is  independent  of  their  representation  in 
any  corresponding  Value  Domain  .  A  specific  Value  Meaning  may  have  more  than  one 


367  A  data  element  is  a  basic  unit  of  data  of  interest  to  an  organization.  It  is  a  unit  of  data  defined,  identified,  represented,  and  conatins 
permissible  values  specified  by  means  of  a  set  of  attributes  [IS079c], 

36S  Data  Element  Concepts  is  a  concept  represented  in  the  form  of  a  data  element,  and  described  independently  of  any  specific  represen¬ 
tation.  It  may  have  zero  or  one  Object  Class  and  zero  or  one  Property  [IS079c]. 

369 

A  Unit  of  Measure  is  the  unit  associated  when  Data  Element  values  are  specified.  The  Unit  of  Measure  Name  designates  the  unit. 
When  specified  the  unit  must  maintain  consistency  with  any  dimensionality  specified  in  the  corresponding  Conceptual  Domain.  Option¬ 
ally,  a  Unit  of  Measure  may  have  specifications  for  the  number  of  decimal  spaces  supported  in  the  associated  Data  Element  values.  The 
precision  is  a  default  tha  may  be  overridden  for  any  particular  data  element  [IS079c]. 

370 

A  Value  Domain  is  a  key  component  of  a  Conceptual  Domain,  providing  representation  for  the  Conceptual  Domain.  An  example  of 
a  Value  Domain  is  ISO  3 166  describing  the  codes  of  the  representation  of  countries  within  a  set  of  seven  Value  Domains:  short  English 
name,  official  English  name,  short  French  name,  official  French  name,  apha-2  code,  apha-3  code,  and  numeric  code  [IS079c], 
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means  of  representation  by  Permissible  Values371 ,  from  a  distinct  Enumerated  Value  Do- 
main  .  A  Value  Domain  may  be  associated  with  other  Value  Domains  through  a  Value 
Domain  Relationship ,  either  as  a  composition  with  other  Conceptual  Domains,  or  as  a 
component  member  of  a  larger  Conceptual  Domain,  using  the  value  domain  relationship 
type  description  [IS079c]. 


Figure  J-9.  Conceptual  And  Value  Domain  Metamodel  Region  (From  [IS079c]) 


A  permissible  value  is  an  expression  of  a  Value  Meaning  within  an  Enumerated  Value  Domain  and  one  of  a  set  of  values  comprising 

an  Enumerated  Value  Domain.  Each  permissible  value  is  associated  with  a  value  meaning  [IS079c]. 

372 

An  Enumerated  Value  Domain  is  an  expression  of  an  explicit  set  of  two  or  more  permissible  values,  and  a  Non-Enumerated  Value 
Domain  is  a  value  domain  expressed  by  a  description,  specification  or  rule,  rather  than  a  set  of  permissible  values.  Both  the  Enumerated 
Value  Domain  and  the  Non-Enumerated  Value  Domain  are  a  sub-type  of  the  value  domain  and  inherit  the  attributes  and  relationships  of 
the  value  domain  [IS079c], 
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J.  DMRR  DATA  ELEMENT  METAMODEL  REGION 

The  Data  Element  metamodel  region,  shown  in  Figure  J- 10,  addresses  the  admini¬ 
stration  of  data  elements.  Data  Elements  provide  the  formal  representation  of  infonnation 
(e.g.,  a  fact,  a  preposition,  an  observation)  about  some  concrete  or  abstract  thing.  Data 
Elements  are  reusable  and  sharable  representations  of  Data  Element  Concepts.  When  a 
Data  Element  Concept  receives  a  value,  a  Data  Element  fonns.  A  Data  Element  is  the  as¬ 
sociation  among  a  Data  Element  Concept,  a  Value  Domain,  and  a  Representation  Class 
[IS079c], 


Figure  J-10.  BMDS  DMRR  Data  Element  Metamodel  Region  (From  [IS079c]) 
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The  Data  Element  to  Representation  Class  association  may  occur  directly  or  by  way 
of  the  Value  domain.  Registration  of  a  Data  Element  as  an  Administered  Item  requires  as¬ 
sociation  with  a  Data  Element  Concept  and  a  Value  Domain.  A  representation  class  quali¬ 
fier,  if  specified,  qualifies  the  name  of  the  data  element.  Data  Element  Precision  specifies 
the  number  of  decimal  spaces  permitted  by  any  associated  data  element  values.  Unless 
specified  differently,  the  unit  of  measure  precision  from  the  associated  Value  Domain  ap- 
plies.  A  data  element  may  also  have  Data  Element  Examples  and  a  Derivation  Rule 
[IS079c]. 

A  Data  Element  may  have  associations  with  several  Value  Domains,  resulting  in  a 
different  Data  Element  Concept  for  each  association.  The  Value  Domain  provides  repre¬ 
sentation,  but  no  links  to  the  Data  Element  Concept  values,  or  what  the  values  mean.  A 
Value  Domain  may  associate  with  multiple  Data  Elements.  The  Representation  Class  is 
the  Classification  Scheme  for  representation,  supporting  a  set  of  classes  to  distinguish 
among  the  various  elements  in  the  registry.  Objectives  of  the  Representation  Class  include 
providing  a  complete,  discrete  set  of  high-level  definitions  for  data  element/value  domain 
categorization,  supporting  the  integration  of  business  rules  for  applications.  Representa¬ 
tional  Class  use  enhances  semantic  control  over  contents  of  value  domains,  supporting  the 
development  of  rules  for  representation  classes  allowing  the  enforcement  of  content  within 
and  among  domain  values  [IS079c].  [IS079e]  and  [IS079c]  provide  an  informational  list 
or  representational  class  terms. 

K.  DMRR  CONSOLIDATED  METAMODEL 

A  consolidated  metamodel  in  Figure  J-l  1  illustrates  the  combined  Data  Element 
Concept,  Data  Element,  Conceptual  Domain,  and  Value  Domain  of  the  [IS079c]  Model. 
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Data  Element  Examples  provide  representative  samples  of  the  Data  Element  [lS079e], 

374 

A  Derivation  Rule  specifies  a  derivation  for  the  Data  Element  [IS079c]. 

375 

A  Representation  Class  is  a  mechanism  for  conveying  the  functional  and/or  presentational  category  of  an  item  to  a  user  [IS079c], 
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Figure  J-l  1.  BMDS  DMRR  Consolidated  Metamodel  (From  [IS079c]) 
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